Bad Timing

A cow-orker of mine once bragged about the DateTime class he’d written.  “It’s fantastic,” he said, “It handles dates from 10,000 BC to 10K AD, down to the millisecond. It does all the correct calendar conversions.  It knows about September, 1752. We don’t have to worry about our time API now.” He left the company a couple months later, and we shipped and started getting bugs from customers.

Most of the features he’d written were great on paper, but he’d never written effective unit tests. The code had a bottleneck that funneled everything through Unix time functions. Nothing prior to 1970 worked, nor anything past the Day of Armageddon in 2038. There was code to handle leap years, but it failed every 100 years.

I rolled up my sleeves and got into it. It was surprisingly hard to get it right. I spent about two weeks writing unit tests and making the thing work as advertised. Our start-up had a really neat smart messaging protocol with date/time as a basic type — we figured that since so many of our customers were getting it wrong, that we’d get it right and leverage that as a feature.

Something about “Those whom the Gods destroy they first make proud,” right. This is a class of problem that everyone knows is trivial. “Of course I know how to do this,” you probably think to yourself, “This is trivial, I can just code this thing up and have an early lunch.” Trivial is the most dangerous word in all of computerdom, it’s a misprounciation of “Evil”. Your blood should run cold if you hear it spoken out loud, because the Gods will be listening carefully and taking notes on how to screw you if they ever hear you say it.

When you deal with time, you have a number of choices.

  • What is the range of times you handle? Computer-era scale? Human (historical) scale? Geologic scale? Ridiculous death of the universe ranges, when protons have decayed, all galaxy-sized black holes have utterly evaporated and even Twinkies have gone a little stale?
  • What resolution do you need? Particle accelerators probably need jillionths of femtoseconds, for instance. Computer-scale times are probably on the order of hundreds of nanoseconds. Human-perceptible and most real-time processing can be done in milliseconds. Geologists don’t give a shit about a stretch of time less than a hundred thousand years.
  • What about daylight savings time? Time zones? Leap seconds? All the people who get this stuff wrong in the real world?
  • Do you need the ability to compare times? (This becomes an issue when you deal with some other time formats, for instance OLE uses floating point numbers, days since some epoch, and fractions of a day, and so representing 10AM exactly is impossible).

That’s just time; we’re not even getting into “date” issues like calendars, or holidays. Holidays are serial nightmares on roller skates.

I believe that most programmers (including this one) do not really understand time. It gets worse, though, when you have to deal with a hardware clock that was designed by a hardware engineer who not only did not understand time, but also failed to devise a way to reliably extract it from the circuit he designed.

“You set the Freeze register and wait, then read the value.”

“How long do I wait?”

“Um . . . about, well, a while.”

“How long is a while, given that this is the hardware that’s supposed to be telling me how much time is passing?”

“I guess you can just delay or something.”

So one of us is going to Hell because we have to turn off interrupts and spin for half an eternity, just to find out what time it isn’t any more. That tick you were interested in? Sorry, it happened while we were busy waiting for some moron of a flip-flop to make up its mind. See you last year.

Recommended reading: Calendrical Calculations. This is a really great treatment of time and calendars, and if you are writing any code in this area you should definitely read the first few chapters and then crawl under a desk and quiver, wondering how anybody gets this stuff right, ever.

Oh, and personal to you-know-who-you-are: I hope you are writing some bloody unit tests now. Right? Okay?

This entry was posted in Uncategorized. Bookmark the permalink.

31 Responses to Bad Timing

  1. Mitchell Perilstein says:

    Certain Sun systems had a great bug in Solaris 8, exactly with that freeze period reading from a time-of-day register. It seems that the OS was able to read while the hardware was in the middle of updating that register to get to a new hour. You would ask for the date and it was possible you’d get the old minutes and the new hour, eg: 12:59:59, 13:59:59, 13:00:00, 13:00:01… so you got +1 hours, then -2 sec. We called that little dance maneuver a clock jump, and our human safety software just loved it when the OS did that. Google for “todds1287″.

  2. vovikb50 says:

    You definitely felt the problem first-hand… Hang in there, man. Doomsday is near!
    Another recommended reading (before Armageddon): “Clock of The Long Now” by S.Brand

  3. Anonymous says:

    Don’t forget about satellites where time itself moves more quickly.

    • Anon says:

      More slowly – relativity !

    • Tobias says:

      change “more quickly” to slower and you have it right.
      Yes time is difficult at times ;)

    • Glenn says:

      Actually, it’s complicated. For a fast moving satellite the special relativistic effects will probably dominate, and ‘more slowly’ is correct. If the satellite is e.g. geostationary, however the clocks will run more quickly than for you on the ground below due to general relativistic effects.

  4. Miguel Farah says:

    “Holidays are serial nightmares on roller skates.”

    DO tell me. :-)

    Back in 2005, I had to write a Java class that would compute whether any given date would be a holiday or not (assumed to be in my country, Chile) to replace the previous one, a shitty class that only considered day+month (which was disastrous for singular, as opposed to recurring, holidays). Oh, boy… it seemed easy enough, but I ran into a bit of a problem: since I had decided to fully support day+month+year, now I had to know *since when* all the country’s holidays are so (and… take into account extinct ones, too).

    I could have gone the easy route and state “Valid from 2001 onwards.”, but I didn’t want to do that. After all, the information I need will be easily obtainable with a couple Google searches, right?

    WRONG.

    There simply was no information available at all, other than perhaps a non-updated webpage listing the holidays for the previous year or something. In the end, *I* did a full investigation (looking for and reading old laws and government decrees) on the history on Chilean holidays, one that only now I’m finishing.

    As a public service, thinking of what I felt when I couldn’t find any information, I chose to publish what I had on my website. I’ve written about 75,000 words on it, plus about 22,000 more on “national days”, enough for an actual book.

    You might want to check out http://www.farah.cl/Feriados/ and http://www.farah.cl/Feriados/DiasNacionales.html (it’s in Spanish, though).

    Oh, and that Java class? I have a good-enough version, but the one that incorporates ALL the information I published keeps getting delayed…

  5. Mike says:

    Are you talking about C?

    Java has GregorianCalendar class already defined.

  6. landon says:

    @Mike: C and C++ mostly. Though we had to do interop with Java, Visual Basic and (shudder) PowerBuilder.

  7. vsync says:

    Good read. I’ve had sort of the corollary to this situation where the client is the one saying “just store the time, how hard can it be”. Time’s one of those problem domains where you have to consider the situations where the data involved will be used, and if you expand the scope you need to reconsider your data formats and algorithms. Way too many gotchas and time is probably a leading contender for the average person’s list of “things in life that are really simple”.

    Erik Naggum wrote a great article called The Long, Painful History of Time, worth reading.

  8. tony goddard says:

    I worked in Saudi Arabia once, and found out that they have a ‘hejira calendar’ based on
    the time which has elapsed since the Prophet Mohamed left Makkah to go to Madina (the hejira).
    This is a lunar calendar, and the year is about 336 days long. The month start and
    end times are determined by the observation of the first sliver of the moon, just after
    the new moon, and the observation has to be made by a respected religious scholar in
    the preceincts of Makkah.

    The calculation of holidays is very important when it comes to the start and end of the fasting month of Ramadan.

    Iran also has a hejira calendar, but dates are rather different from Saudi dates.
    No one in a USA based company should even attempt calculate this, since selling any hardware / software to most Iranian organisations is prohibited under USA law !

    twitter: @d4maths

  9. J. says:

    Just to illustrate the point about holidays, here in Germany many of the holidays depend on the federal state (Bundesland) you are in, especially the lesser religious ones. For example, assumption is a public holiday in the Saarland and in parts of Bavaria (don’t ask me which parts, or even how to find out).

  10. Luke G. says:

    Wow, what an eye-opening read! I’ll admit that I would have been one of *those* people who considered the handling of date/time to be a fairly simple one. Now I know better! ;)

    @Miguel Farah
    If you have enough data to write a book…why don’t you? Self-publishing via electronic media is…(I know, I hate to say it)…almost trivial these days. You could put a self-published book on Amazon, for instance. Just a thought.

  11. Tim says:

    Hi,
    We use Copp Clark holiday files which are updated to account for holidays here. More reliable that inconsistant rules.
    Tim

  12. Derek Dongray says:

    A few years ago, I wrote an SQL function to calculate holidays for multiple countries that we dealt with at the time. I identified 8 types of public holiday:

    1. Relative to Easter Sunday – e.g. Ascension Day, Good Friday.
    2. Same as above bu relative to Greek Orthodox Easter Sunday.
    3. Fixed date every year – e.g. 4th July
    4. As 3, but gets a replacement if it’s at a weekend, i.e. if it falls on Saturday or Sunday then Monday is a holiday – e.g. Christmas Day in UK.
    5. As 3, but is the 2nd day of a pair of days, so may also fall on a Tuesday – UK Boxing Day is the only example I found.
    6. Specific day of a week in a month – e.g. first Monday in May (May Day), last Monday in August (UK Summer Bank Holiday)
    7. Fixed date in Hijri (hejira) calendar. Fortunate MS SQL has a Hijri datetime which uses the Kuwaiti approximation, which is mostly correct, so you can avoid calculating phases of the moon and accessing weather reports.
    8. Relative to winter solstice – e.g. Chinese Qing Ming Je.

    Of course, as the function grew in complexity and started to indicate that it needed astronomy in addition to religious calculations to decide if a date was a holiday, I did wonder whether it would have been simpler to create a table and just hard-code every public holiday for a few decades!

  13. caroza says:

    Batch schedules, and the number of ways in which you can get 1.24.09 am on 22/5 to miraculously turn out earlier than 22.23.47 on 21/5….and getting fin year ends into yyyy-mm formats without subtracting a year when you shouldn’t…oh, and BI events for slowly changing dimensions where the source system doesn’t give you any more than last update date (so you try to change records which don’t exist yet…!). Anybody remember the COBOL programmer who had himself put into suspended animation so he wouldn’t have to work over Y2K? He got woken up a little late due to a glitch. Someone opened his capsule and said, “oh, thank heavens, you’re a COBOL programmer, aren’t you?”. “Yes”, he said, “but why do you want to know?” “Well”, she replied, ” it’s the year 9999 and we’ve got the year 10 000 coming up….”

  14. austin says:

    wow, that is a pretty dumb calendar. but i wont get into that…

    “You set the Freeze register and wait, then read the value.”
    i thought computers used a crystal oscillator with interrupts at some fixed interval. if you dont want to use OS level time structures can’t you just listen for that interrupt?

  15. Gary says:

    … ‘“Of course I know how to do this,” you probably think to yourself, “This is trivial’ …
    No way! I’ve been writing code way too long to think that. There’s nothing trivial about date/time/datetime/etc/etc – great article! Thanks for the chuckles :)

  16. Hal says:

    Try Julian Barbour’s “The End of Time”. We all know there is no absolute time (thank you Albert Einstein). Barbour goes further: Time doesn’t exist.

  17. Miguel Farah says:

    @J. : Yeah, one of the things that simplified my life is that in Chile there are *currently* no regional recurring holidays. There have been a few single ones lately, but people just ignore them or treat them as exceptions. In the past, however, regional holidays were common (and a good part of the reason my software project is still unfinished: I’m still heavily researching the history of my country’s territorial organization – it’s much more complex than could be initially expected for a unitary 200-year-old country).

    @Luke G. : I’ve thought about it, but not much. First, I want to *finish* the research, then I’ll worry about publishing it in something other than a simple webpage.

  18. Miguel Farah says:

    @Derek : in my case, Chile, there are two extant holidays that depend on Easter and four extinct ones, all calculated as a certain amount of days before or after Easter. What’s really fun is that there is *no law* that mandates how to calculate said date – everyone’s simply followed the custom that stems from the time Church and state hadn’t been separated yet (this happened in 1925). This implies that at some point, someone from another Christian church could sue the state to force the Easter holidays to fall on another date.

    And all hell will break loose if a recently introduced bill that creates two holidays for Muslims is approved: those two would follow the Islamic calendar… while people around here already find it difficult to grasp the intricacies of the Gregorian Calendar.

    Yeah, “serial nightmares on roller skates”, indeed. I’m just happy I don’t have to take into account more than one country.

    I *do* have to say one more thing: “I did wonder whether it would have been simpler to create a table and just hard-code every public holiday for a few decades!”. I saw a certain place that did exactly that – the table wasn’t properly maintained and within three years it was full of errors that extended both into the future and the past. IMNAAHO, it’s better to suffer through the pain of properly programming a set of computed rules (a Java class, in my case) than to hard-code everything and just do a lookup: the knowledge of *why* a particular date should be a holiday will be lost.

  19. Jon Skeet says:

    Thanks for validating that all this stuff is painful – as if I needed to be told that ;)

    I’m currently working on the Noda Time library for .NET, because dates and times in .NET aren’t handled terribly well. And yes, it’s a pain. You didn’t even mention the localization – handling cases where some countries use different words for months depending on whether the day is being specified or not, and how some countries don’t have AM/PM designators etc.

    Have just ordered Calendrical Calculations – it’ll be interesting to see how much I’ve already got wrong in Noda Time :)

  20. Aaron Davies says:

    I once got an email at work (I was working in Singapore at the time) saying that the Indian authorities had moved Diwali for that year after redoing their astronomical observations.

    @Derek Dongray: did your system allow for “2nd day of a pair of days” relative to “specific day of a week in a month”? An example would be election day in America (not precisely a “holiday”, but I’m sure there are contexts where it’d be useful to define it as such), which is the Tuesday after the first Monday in November (in even years, if we’re only considering national elections, though most odd year local elections are scheduled the same way).

    • Miguel Farah says:

      @Aaron In Chile, election days *are* holidays since 1988. In 2009, however, they were moved to Sundays, so the effect is nullified (but the law still says they’re holidays).

      Now, “Tuesday after the first Monday in November”… wouldn’t it be the first Tuesday in November?

  21. Mike Albaugh says:

    Anybody remember VMS, and the handy batch file to “Spring Ahead” and “Fall Back”?
    Unlike Unix (but like Windows?) VMS keeps (kept?) local time, rather than UTC with an offset. This could have “consequences” for the equivalent of chron jobs, but it was possible to trim the clock frequency. The batch file would “speed up” the clock starting around midnight (IIRC) so that by the time the switch from 2:00AM to 3:00AM needed to be complete, it was. Likewise slowing down in October. An interesting solution to a problem that didn’t have to exist.

    OTOH, I (not so) fondly recall having to use Outlook, which got its time-zone offsets twisted pretty frequently, at least in regard to conference calls scheduled over multiple time-zones. (And then there are the Linux distros who decided that
    having _one_ place for the timezone info across all versions of all distros was just too authoritarian, so I had to play “find the TZ files that this system _really_ uses” every time Congress ran out of stuff to mess with and changed Daylight savings again.)

    As long as we’re posting links, I refer to:

    http://h71000.www7.hp.com/openvms/products/year-2000/leap.html

    responding to a bug-report about VMS counting 2000 as a leap-year. (Note that IIRC, the last Windows patch for leap-year in 2000 went out in Nov. 1999)

    Maybe we need to follow the path of the timing-designer from Soul of a New Machine:
    “…I will deal with no unit of time smaller than a season” :-)

  22. BlueRaja says:

    There comes a point in every good programmer’s career when he learns that large dates and times are impossibly difficult to work with. Thank you for sharing yours, Landon, it was a good read :)

  23. joel garry says:

    You can google LA Schools 2011-2012 calendar, and see several carefully composed overlaid calendars for the various different tracks schools within the district have.

    What it doesn’t show is that, based on state revenues, it might change, getting more furlough days. Or it might not, the district will decide in the middle of the year.

  24. Wally The Walrus says:

    I’ve been through the date and time hell for embedded systems. Thats when you find that different C library vendors do different evil things, many of the C and unix runtimes can’t be relied in all circumstances – and watch out what your embedded C compilers and runtimes do.

    The while nightmare took about 6 weeks to sort out – and at the end of that the big boss marked me down in performance-rating land because he thought it should have been trivial and done by somebody more juniour (and with no idea…. like the big boss).

    The strangest demands come up: We want you to make a box that goes in an equipment rack. No UI, no knobs, no dials, no buttons. It needs to transmit date and time. It also needs to know about public holidays and daylight saving time. But you can’t send that information in to the box in any form – it must be there when it is commissioned.

    Fine and dandy you think. But how, oh how, does it handle the fact that public holidays and daylight saving start/end time are regularly changed by Act of Parliament?

  25. Xavier Flix says:

    For the final University project (circa ’94) we created a device which performed some extremely complicated processes using a quite powerful Intel-based micro controller, over 2 dozen multiplexers, as many latches, lots of memory and a three-row dot matrix display… well, it was quite a complicated device that, if it worked, would substitute one designed by Honeywell and would be used on a very important local factory.

    So the day of our presentation arrives, we show off our device, which looked quite impressive inside a transparent plexiglass box. It actually looked something like the Mac Cube, but transparent.
    Everything worked perfectly and everyone realized how of an improvement over the old device it was.
    But, as we were almost ready to finish the presentation, some woman in the back stands up and asks: and where’s the date on the display?

    We explained that the date display on the old device was more of a space-filler and that it wasn’t required for the worker to utilize the device for its actual purpose.
    As she sat down, she mumbled: well, I guess the “old one” (with a sarcastic voice) is still superior.

    So I (the programmer of the team) had to sat down and develop a date and time calculation process!
    It took me about a month to get it right and in the end, the code amounted for 35% of the whole program that ran in the micro controller.

    From that day, I just refuse to do date/time calculations unless the framework on which I’m working supports it!

    PS: Great read! I took me back ;)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>