What Time Is It?

Let me give you a scenario.  You are having some problems on the network that are spread across several devices.  You go into the log file of each device and see the following in each of the logs.

Now looking at the first image, it’s easy enough to see what I did, I just shut down all of the interfaces on my switch which was in turn connected to one or more interfaces on each of the other devices.  What I want to draw your attention to though is the timestamp next to each log entry.  If you notice, not one of them matches another one, and none of them match the time on my computer.  Now reading these logs is easy because I cleared them all just before I did this but in the real world, there would be hundreds if not thousands of other entries surrounding the one or two entries that we are interested in making it extremely hard to find what we are interested in.

 

So why did this happen?  Well, the short answer is that none of my clocks (with the exception of my computer) is actually synchronized.  Because the clocks aren’t synchronized, they show whatever random time it happens to have, which begins to drift further and further away from the true time.

So how do we fix this?  Well anyone who has been around IT for more than a few days knows about Network Time Protocol (NTP) but how well do we understand it and more importantly, do we actually use it.  The current NTP standard is NTPv4 which was adopted in 2014.  While the accuracy of NTP is partially dependent on the network conditions between the host and destination, NTPv4 is designed to be accurate to within 10s of milliseconds even under less than ideal conditions.  This means that all devices synched to an NTP server should match each other almost exactly.

NTP is configured by default on the WIN-T baseline configs, but it is frequently not actually implemented correctly.  So why is this?  Well for two primary reasons.  First, by default, the WIN-T network uses the RHN/THN as its primary source for NTP time using the TDMA tunnel address (172.x.x.1) as the source.  This is fine assuming that the RHN/THN is actually providing NTP services and that their time is accurate.  Just based on my own personal experience, this is not always true.  The second reason why NTP often doesn’t synch is because of a large discrepancy in what the router is currently tracking as it’s time vs. the time that is being reported by the NTP server.

By default, a router will only sync to an NTP server when their times are within 4000 seconds (just over an hour) of each other.  If the times fall outside of that difference, the packets received from the NTP server are ignored and the time is not adjusted.  This rule is waived when you initially implement NTP and when the router reloads however if that node doesn’t have WAN connectivity at the time of reload, than it is possible that the time difference is greater than 4000 seconds and the packets are being ignored.  The easiest way to correct this issue is simply to manually set the clock to the approximate correct time using the command set clock.  Once this occurs, the router’s clock will fall within the window of the NTP server’s and the two devices will sync up.

Why Timing is Important

So why is timing important.  Well, as I said above when it comes to troubleshooting the network, being able to see the sequence of events as the move across the network.  This can help us understand what is going on and aid in troubleshooting.  More importantly though, having accurate log files with correct time can be critical in responding to a security incident where we must be able to see an attacker move across the network.

I have heard some people talk about how NTP is what is responsible for the timing on our HCLOS links and that fixing NTP will fix the timing problems we often see on these links.  While it is true, we do get the timing for those links off of the local GPS inside the JNN, it is a separate clock signal, not NTP.  Fixing NTP will not fix the timing on the HCLOS links.

JNN NTP Architecture

Now that we have figured how to get NTP working, it’s time to talk about how most effectively to deploy it on the WIN-T network.  As stated above, the JNN’s Tier 2 routers (both SIPR and NIPR) are configured to point to the RHN/THN router via its tunnel interface.  From there each subordinate CPN Tier 2 Router points back to both JNN’s tier 2 routers.  All switches, firewalls, and other devices look to the local Tier 2 Router at act as its NTP server.  With this setup, ever device directly or indirectly gets it’s time set based on the time on the two JNN’s Tier 2 Routers.

Because the NTP time provided by the RHN/THN may or may not be configured (or accurate), it is important to look for other sources of time.  The NIPR side is actually extremely easy because each JNN has a GPS clock installed to provide timing for the serial links.  The NT2R can easily be configured to look at the local GPS for a true authoritative time, but this is not possible on SIPR so we have to keep looking.  The Naval observatory operates two sites (Washington, DC and Schriever AFB, CO) that provide authoritative GPS time on both SIPR and NIPR.  These time sources are restricted to military users on both sides and because they currently run NTP v4, they provide authenticated timing to meet the current  STIG requirements.

Non-authenticated time can be obtained simply by pointing your routers to their NTP servers, but a request for an individual unique authentication key can be obtained through the use of a simple signed email from your work account.  Instructions on how to get these keys (along with the server IP addresses) can be found on their website.  What you receive back will be an email with the authentication key and instructions similar to the ones shown below.  Place the configurations on each of your JNN’s Tier 2 Routers and you will be good to go.  Your Tier 2 Routers will synch to the naval clock which will in turn sync the Tier 2 Routers for each of your CPNs which will in turn sync all of the other devices at each node.

ntp authenticate
ntp authentication-key 276 md5 <key>
ntp trusted-key <key#>
ntp server ntp.usno.navy.mil key <key#>
ntp server ntp.usnogps.navy.mil key <key#>

The final step in this is to ensure that the management computers in NETOPS and at each JNN/CPN are also configured to point to the local Tier 2 Router for NTP services.  This will guarantee that all devices have a common time across the entire Brigade WIN-T network and that when you review log files, they will look like this instead of what we saw above.

What Time to Use

The last thing to talk about when are talking about time is just what time to use.  Generally within the military we operate under either Zulu time (Greenwich Mean Time) or the Local time.  There are advantages to both.  Local time is easy to understand because you likely have your watch set to it, work your shift based on it, and the sun goes up and down each day based on it.  The problem with local time though is that every time you go somewhere new (deploy, go to NTC, or whatever) you have to change the timezone that you are operating off of which can be a pain when you have to change that time on a huge number of devices.

Zulu time on the other hand doesn’t change regardless of where in the world you are.  It is universally understood by pretty much anyone in the military, and there is no time change twice a year involved.  On the other hand, a lot of units don’t operate off of Zulu time on a day to day basis.  One thing to remember is that the router always operates on Zulu time (assuming you have it getting NTP updates), but we can setup timezones to display the local time using the clock timezone command.  Additionally, you can have the routers automatically adjust itself for daylight savings time using the clock summer-time command.