Showing posts with label troubleshooting. Show all posts
Showing posts with label troubleshooting. Show all posts

Friday, 22 September 2017

Clock debug log #1 - month rollover issue

Here I'm going to talk about a month rollover bug that my clock had been experiencing.

In a previous post I spoke about an issue that came about when I showed the clock to a friend. When the clock advanced onwards from 23:59:59 on, say, the 31st day of a month to what should have been 00:00:00 on the 1st day of the following month, it actually advanced to day 0 - which is something that wasn't supposed to be able to happen.

I'd designed in a preset input to the first stage of the day-of-month units counter such that when ever certain reset signals were asserted to reset the counter back to the start of the month, the units counter should have been pre-loaded with a 1.

But I've already discussed and fixed that issue. This post pertains to a different rollover bug that I encountered after I had debugged and fixed the one above.

And the story behind this one is that when the clock is on day 29 of a month, and it needs to advance to day 30, it would actually advance to the next month and again to day 0, instead of the 30th.

My suspicions about why this was happening surrounds the use of ripple counters in my clock. I suspected that because of the way ripple counters work, in that it can take "some time" for a change starting at the least significant bit to make its way through all of the stages of the counter. What this means is that for a very brief period of time the counter can represent a number of different values in quick succession as the flip flops transition between different states.

And because of this, I figured that the block of combinational logic that determines what day of the month it is, and based on the "length-of-month" input (30 day, 31 day or February), at some point in time the state of the counters was such that the reset logic figured that the date actually needed to advance to the next month instead of letting the counters advance to the next day.

To test this theory I needed to probe around the reset circuitry in the clock to see what it was doing. Specifically I was looking at the output of GATE68 in the calendar module. The output of this gate will be high when the day of month counter should reset and advance to the next month.

I set up my oscilloscope to trigger on a high going edge of the output of this gate, set the clock to 23:59 on Jan 29 and then let it run. The sequence of events is as illustrated below.


As you can see (although the scope probe wasn't connected in these photos) that once the clock advances past 23:59:59 (to what would technically be 24:00:00) the reset logic generates a pulse which causes the day of month counter to reset to 00 and the month counter to advance to the next month.

So I thought about it for some time, and I reckoned all that might be needed would be to add a small capacitor to this reset signal, such that while the ripple counters are sorting out their final state, if the reset logic should happen to generate a pulse during this period, the initial bulk of that pulse should be absorbed by the capacitor such that only once the ripple counters settle down, and if their state represents something which should cause a reset, then the reset signal will remain high for a significantly longer period of time - long enough to fully charge the capacitor and thus allow the reset signal to actually cause a reset.

So I tried this out:


In this photo you see a little grey probe sticking out the left hand side of the clock. This is connected to the output of GATE145 in fact. The orange wire connects to a 1nF capacitor, the brown wire connects to VSS (ground), and the green wire connects to my scope.

As you can see on the face of the clock, this did indeed seem to do the trick, as the calendar had advanced correctly from the 29th to the 30th and remained within the same month, instead of moving to the next month.

I do still see a small rise in the reset signal, I guess 1nF is only just enough to cover this, but at about 1.5V its not high enough to drive any MOSFETs and cause a reset.


For completeness, heres a photo of the newest addition to the clock (the little orange blob), tucked away all nice and cozy in there:


The schematic for the calendar module has also been updated to reflect this recent addition. And with that, I am no longer working on any issues - the clock works and functions as it should. Only time will tell if anything else pops its head up.

Wednesday, 16 August 2017

Clock design and build log #6 - calendar decoders

I'm going to do another combo post here, simply because the design portion is quite straight forward.

And if I'm honest, this was probably the most stressful module to build in the entire clock, namely because the month name decoder is such a monster - if there was something wrong with the design, or some issue with my translation from logicly to EAGLE which then made its way in to the physical build, then it was going to likely be quite painful to fix.

The design work for this board is mostly covered in prior posts about 7 and 16 segment decoders, namely these ones:


Most of the additional design work was to take the logicly circuits and re-build them as EAGLE schematics, since all of my builds have been guided by such schematics (logicly was just for simulation and verification).

As you'll see when you look at the schematics linked below, theres quite a lot going on in the month name decoder. It has a lot of gates which are connected together and was going to make for a challenging build.

The day of week decoder is quite tame in comparison, and the decoders for day of month have been built before on the time decoder module and so are tried and tested, and actually were very simple to put together.

So, the build. As I was expecting a fairly dense board for this build I decided to give myself as much room as possible and started laying out my transistors right up against the bus connectors at the bottom of the board. This made things a little interesting when connecting the various output gates to the bus, but overall it wasn't too difficult.

The first decoder to be built was the day of week, which would display something like MO, TU, WE etc. This build, as anticipated, went very smooth, and there were no issues encountered - it just worked. Phew - one down, two to go.

With the partially built decoder board in the clock I cycled through each of the 7 days of the week using the day of week set button on the calendar module and was pleasantly surprised to see all of the short names reading out as I had designed them. Here's a small selection:


Next were the day of month decoders. I'd already built these kinds of decoders before, and I didn't expect to hit any snags with these either, and nor did I. Two down, one to go, and so far everything was running fine giving me this:


Once again I cycled through all possibilities to ensure the correct numbers and sequence were displayed, and I also cycled through the various months to ensure that the counters would reset and count the correct number of days per month, this also proved to work perfectly.

I was feeling slightly more confident now that I was about to head in to what I felt would be the most difficult part of the build, the month name decoder. I figured the best way to go about it was just start doing it, and deal with the issues as they came up. To help, I would build the decoder "letter by letter", and use the logic analyser of my scope to check the outputs (8 at a time) were in the correct states for each month of the year. Heres a glimpse of how I went through that process:


Power and inputs to the decoder board in this case were supplied by a battery and some wires off a breadboard. I thought it would be easier if I had to make corrections, and didn't have to keep building up and tearing down the stack of boards.

Through this process I did happen to come across some mixed up connections which had occurred during my translation from locigly to an EAGLE schematic. It was a matter of buzzing out the suspect signal wires with my multimeter, working out what had gone wrong, correcting the schematic and then re-soldering wires to the correct topology.

I kept at it and eventually I got to the point where I soldered (what should have been) the final 10 transistors!


And after wiring it all together and testing the outputs with my logic analyser I was at a point where I wanted to plug it in and test it with the display board instead of looking at high and low signals on a scope!

And, after plugging it in I was greeted with a most welcome sight (after first closing my eyes and psyching myself up to open them, hoping to see the right output)!


Or so I thought, until I moved the month along to February.....


Hmm, that's not right! This turned out to be a very simple fix, however. After verifying that the calendar module was indeed outputting the correct signals to the correct pins, I tracked down and corrected two crossed wires on the display board. That fixed it right up, and cycling through all of the months I got a perfect readout for all of the names - just as designed in a spreadsheet all those months ago!

AMAZING I'm thinking - it's pretty well DONE.

It was at this point that I showed the clock to a friend, and after attempting to demonstrate the month rollover where the day should reset to 01 instead of 00 that I discovered the issue where it actually rolled over to 00. That was then corrected (or so I thought) with a little additional circuitry.

But I have since discovered another issue. On July 29 when the date should have incremented to July 30 it actually rolled over to August 00. Ahhhhh crap. So I'm currently pondering the solution to this, and I think I know why it is happening, and what the fix will be. Stay tuned for a later update about that (I really hope this isn't going to become a recurring theme.......) 😔

Anyway, it's actually very useable as a clock and calendar at this point, despite some niggling issues which I will hopefully resolve in time.

Here is the end result for this board:


As always, schematics are linked here:


And with that, it's basically done now. I plan to write a "debrief" post and talk about things that went well, what didn't, and my general thoughts of the project. Stay tuned for that in the near future.

As for the fix for the rollover issue, I will get to it in time. After such a massive project I want to take some time to do a lot of stuff that I didn't do while I was doing this, and basically do anything but this for a while! The display dimming board will also come at some stage in the future as a result.

Sunday, 30 July 2017

Clock build log #5 - calendar module

One thing I've come to realise throughout this project is that describing the build is reasonably uneventful - it's really just a repetition of place components and do a load of soldering. It makes for much less of an interesting read.

Certainly on a personal level I have found the troubleshooting process to be much more interesting - discovering an issue and trying to figure out how to fix it in as few changes or additional components as possible. Some of my boards are quite dense with little room for more logic, especially if a fix needed a large block of logic to implement. And as you've seen the "rats nest" of wiring on the rear of the boards actually poses quite an accessibility problem getting to solder joints, even with the reasonably fine 1.5mm soldering iron tip that I have used.

Luckily, however, problems requiring fixes have been reasonably uncommon, with maybe a transistor or two added here and there. And I've been quite thankful for that.

So having said all of that, I'll skip most of the details of the build of this board, and fast forward to the troubleshooting stages.

Like in previous builds involving ripple counters, I built each stage one by one so that I could test that they all worked before burying them under a lot of wires. And like all of the counter stages that came before, I didn't encounter any significant issues. In fact, if I encountered anything that could be considered significant, it was generally that a floating reset input would generally cause that stage to not function until I pulled it low (otherwise it was in constant reset). One of the disadvantages of using MOSFETs over other types of transistors is that their gates are capacitive, which means that when they are floating their state is somewhat non-deterministic - that is to say they could be high, low, or somewhere "kind of in between" - so pulling up or down is essential.

This is often very easy to fix. Using a pair of tweezers I can create a short between the gate pin and the negative power supply rail, and that was usually enough to sort out that gate long enough that I was able to test the functionality of the n-bit counter that had been built up. Some gates were a little more stubborn than others and needed a tiny piece of wire soldered in to hold the pin low. The primary reason that was an issue at all was due to the way I built up the counter logic. I would typically start with the counter stages and then move to the surrounding reset logic, so to begin with the reset inputs of each counter stage were not connected.

By far the most significant issue that I came across with this build was to do with the preset input which was supposed to ensure that when the month counter resets (i.e. when the calendar moves between months) the day of month counter would reset to 1 instead of 0 (since there is no day 0 in any calendar I know of). As you'd see from the design log for the calendar module, I did breadboard the circuitry to determine whether or not using a small capacitor at the appropriate point in the circuit would hold the preset input low for a short period of time which would enable a corresponding reset input to assert itself high prior to the preset input going high. This sequence is required in order for the preset to work, and certainly it worked on my breadboard. Hence I designed in what was to be the original design of the circuit. But it turned out that this still didn't work quite right once the circuit had been fully built up, and it only became apparent when I showed off my clock to some friends and this little gremlin surfaced. So it was back to the drawing board to find a solution.

I started by scoping out the various reset signals to find out what was happening, and unfortunately I forgot to capture an image of the signals on my scope, but the basic problem that I discovered was that the reset signal that is generated by the large block of reset circuitry below the units counter would be so brief that combined with the capacitor loading the preset input I think it just never actually had a chance to go high. I believe I did notice the tiniest of bumps in the trace for the preset input where that signal had started to rise, but by that stage the counters had reset, and the resulting reset signal was no longer asserted, so the preset signal was dragged back down to ground.

My solution involved adding in some more gates to form another SR latch, one input would come from the original inverter that was connected to the preset input, with the capacitor removed. The other input would come from the AC_CLK/ signal to reset the SR latch once the preset input was no longer going to be needed. The capacitor was then moved down to the appropriate output of the SR latch and then fed in to the preset input. I did also breadboard this solution to ensure I wasn't going to be wasting more time, and that also proved to work, only this time in-circuit.


I really do wish I captured a screenshot from my scope so I could visualise all of what I just said above, but you'll just have to believe me (and I hope it all made sense). And yes, you're seeing the finished clock there, before I've gotten around to writing any blog material for the calendar decoder module. Oops? :-)

So while the day of month counter was now resetting correctly upon the month rolling over, the next problem I found was when pressing the reset button at the back of the clock, the day of month would reset to 0 instead of 1. I had a solution to this pretty much already worked out, and it used a typical combination of logic that I have used elsewhere to "suppress" a certain signal when another signal is asserted. This involves inverting the AC_CLK/ signal and feeding it in to a dual input NOR gate, with the other input of that NOR gate connected to MCLR. When ever MCLR happens to be asserted, the AC_CLK/ signal will not propagate to the SR latch to reset it, and thus the preset signal will be able to have some effect.

The result looks a bit more complicated than the original, and I guess it is, but hopefully is easy enough to follow:

Before
      
After

I was fairly confident this would now work properly, so I went straight ahead and built it up - a total of 7 transistors making 4 new gates (146-149). And afterwards when testing it out, it all worked perfectly. Day of month was now resetting correctly upon month rollover and when the reset button was pressed.

Calendar module done! Here are the results:


Since there were some modifications to the circuitry which occurred after the design log was published, I've uploaded revised schematics along with this blog. They are in the same location as before.

Friday, 12 May 2017

Clock build log #1 - analogue board

Alright, alright already! I know this is what some people have been waiting for - actual freakin' construction. So I'll just get straight to it.

Build

I started with construction of the power supply, since that was going to be rather necessary. For a slight retro look I decided to use an axial capacitor as the smoothing capacitor on the input to the linear regulator. The full bridge rectifier has a 100nF capacitor in between the DC terminals as I read this can be good for filtering noise at the output of the rectifier. This then feeds via the smoothing cap in to the 7809 regulator. Another 100nF cap at the output of the regulator helps to filter noise on the output here too (not that I expect that much noise, but it can't hurt) and the positive rail then feeds through a PTC (fuse) for short circuit protection - theres a lot of fiddly wiring all done by hand, and if I happen to short something out or it breaks loose, I don't want anything getting damaged.

It was hard to find a 2.1mm DC jack that wouldn't need me to drill in to my board to create slots that the terminals could be inserted in to for soldering (i.e. nothing with pins on a 0.1" grid), but I had a flash of inspiration and figured that a surface mount jack with some loops of wire over the top of its terminals could be used to make my own "through hole" socket, so thats what I did, and this is what I ended up with for the power supply:



Quite pleased with that result, and the DC jack feels very solid. I used the off cuts of the rectifier diode leads to form the loops as they were quite hefty.

Next came a sprinkling of other building blocks and bits and pieces: the power on reset RC circuit, the AC_CLK/ generator and isolation switch, reset button, and a handful of transistors forming a few gates of logic to glue all of that together.

Soldering all of that in to place, the board now looked like this:


Then I started soldering even more transistors to start building the prescaler.



And around 100 transistors later and after much effort soldering the many many fiddly wires, I was left with this:



The 8 pin socket on the back forms a bus to the next board to carry power and the MCLR, AC_CLK/ and 1PPS_CLK/ signals over.

Confession: construction of the analogue board started in early March, and a couple of things have changed in between then and now, including the AC_CLK/ generator and some circuitry related to the reset button which provides some (better) hardware debouncing. So if you're wondering why perhaps the components you see don't quite line up with the schematics I posted recently, that would explain it. The schematics represent how it has actually been (finally) built.

The thinner blue, red and black wires are 0.22mm2 or 30AWG Kynar insulated wires. They are also sometimes known as "wire wrapping wire".

And here's another slightly artsy shot, because I couldn't help myself.


So yeah, thats about it for construction of the analogue board. The next step was to test it and figure out if this was going to work. And this was where I learned my first lesson: don't build too much all at once.

Testing

Testing the power supply was easy with a multimeter, and the results predictable. 12V AC in, 22V after rectification and smoothing, fed in to the linear regulator which produced 9V DC out.

The power on reset delay circuit was also easily testable by probing the drain pin of its transistor while applying power. When power is applied you see the output high, and after a short period of time while the capacitor charges up it goes low.

That feeds in to the reset mux, along with the reset button, so testing that was a matter of probing the output of the inverter that sits after the NOR gate to which the power on reset delay and reset button are connected. The inverter outputs a positive logic signal referred to as MCLR, and it is high when ever the reset button is pressed, or when the capacitor in the power on reset delay circuit is charging just after power on. That also checked out OK.

Testing the AC_CLK/ generator is a little more complex, I really need to see the waveform it produces alongside the input AC sinewave since it happens too fast to see on a multimeter, so this needed an oscilloscope.

Backstory: I'd been hoping to win something as part of Keysight's Oscilloscope month promotion, but alas I wasn't so lucky. Towards the end I caved and ended up buying myself a Keysight "premium used" MSOX2024A which ended up costing me about half what a brand new model would have cost. I have to say I'm incredibly impressed so far, it looks for all the world like a brand new scope, and came with 5 years warranty. Highly recommend you check out this route first if you're looking to buy a Keysight... So yes, some of the material for this post dates back to then and even earlier. I also started a new job within the past month and a bit, so that has kind of slowed things down a bit too.

Using my newly acquired scope I took some measurements of the rise and fall times of the output of the AC_CLK/ generator to see what its waveform looked like. A very early version of this part of the circuit used a BC547 and I wasn't happy with the output, in particular that the output voltage was only around 4.5V which was uncomfortably close to the gate threshold voltage. So to be safe I re-designed it to use a MOSFET instead, and was much happier with the output voltage at 9V. The rise and fall times were nice and sharp at no more than about 15uS (microseconds) at worst (fall time was just a smidge over 3uS) - also much faster than the original BC547 based circuit. Here are some scope traces of that:



That is of course looking at the time taken to transition between ground and peak voltage - actual "apparent" rise/fall time would be much quicker, because the transistors don't need the full 9V to turn on or off, that happens closer to 4.5V.

The output of the AC_CLK/ generator feeds in to a NOR gate along with the MCLR signal, such that when ever MCLR is asserted the AC_CLK/ signal is held high. When MCLR is not asserted, the AC_CLK/ signal is free to pass through and oscillate. So testing that was a matter of probing the output of the inverter that forms the output of that logic. This was also easy to test with a multimeter in frequency mode, I got ~50hz, so that is all working fine.

On a roll I thought.

Then it came to the prescaler. Was I going to get lucky and was it going to work at all? All of the simulations and breadboarding said it should, but now we're in the real world and at bigger scale.

I used my multimeter in frequency mode to probe the output of the first stage. It was giving me 25hz which is what I expect (since every stage of the prescaler acts as a divide by 2 of its input frequency. The second stage was giving me 12.5hz - brilliant. And the 3rd stage was giving me 6.25hz. "Amazing" I'm thinking, "did I get this all in one go?"

Then I probed the output of the 4th stage and nothing. Zip. Nada. Ah crap, my luck had run out. Since I didn't actually have my oscilloscope at this point in time, I fabricobled a crude logic probe out of an LED, transistor, and a couple of resistors to allow me to get a quick visual from the circuit:


If you would like to build your own, here's the schematic of what I built (you may need to adjust the value of R2, I picked 2.2K to work with the 9V of my power supply, but other logic circuits might need something a little less, perhaps 1K for a 5V circuit for example):


Using this I probed around at the various inputs and outputs to the 4th stage to see where the signal was getting lost, and it just happened to be a solder joint that hadn't properly wetted from the source pin of one transistor to the ground rail. A tiny bit of reflow sorted that right out, and the 4th stage was giving me 3.125hz. Bingo. That was where lesson #1 was learned, because the solder joint was buried in between the wiring, it was a bit tricky getting down in there to make the repair. For future portions of the build I decided I would build and test counter stages one by one to avoid problems like that.

The 5th stage gave me 1.5625hz, and finally somewhere around 0.78125hz at the 6th stage (logical conclusion, because my multimeter can't display that many decimal places). Great, they are all working.

I then proceeded to hook the second, fifth and sixth stages (which represent BCD values of 2+16+32) up to the 3 input NOR gate which feeds the reset circuitry, and probed the non-inverting output of the SR latch, and what do you know, I was getting a very brief pulse once every second.

And with that, the analogue board is complete! It is producing all of the necessary clocking and other signals that are needed to move forward to the next board which will be the "time module", counting seconds, minutes and hours, so I hopefully shouldn't need to pay any more attention to this one.