Super Filter and Synthesizer

Probably you have read my super filter design. It aims to augment projects starting from a setup like pictured below.

The idea is to leverage my clock library to provide a super filter.

Inside this filter uses the library and feeds its results into a DCF77 synthesizer.

Pretty simple and obvious once the hard work of implementing such a library is done. However one of my readers complained that the result performed significantly worse than the library. This appears surprising at first glance but it was actually a design choice which might by argued about. Before this issue can be resolved you need to understand the different internal states of the clock library. These states were introduced with the local clock and are described at this experiment in greater detail.

The point is that the library has an internal state engine that follows the diagram below.

Clock States

Depending on the internal state the library will have more or less accurate time. Synced is most accurate, then comes locked, unlocked and free. Dirty might be accurate but useless is almost never accurate. The point was that I choose to only synthesize the signal if the internal state was “locked” or “synced”. In all other cases I was just passing the signal as it was received. This was to guarantee that the filter would not create “leaps” in its output (except wrongly placed 1s leaps for leap seconds). However the library keeps pretty accurate time even in free mode, especially if it was able to auto tune for a week or so. So the expectation was that even when the signal gets lost the synthesizer should still provide usable output. The impact of this is that the clock might leap an arbitrary amount of time once it resyncs. OK, I am exagerating. It would probably leap less than 1s for each week it was runnning “free”. Still this might be unacceptable for some projects.

I see some reasons why leaps may be unacceptable. One reason are “alarm triggers.” I would suggest to change those to “alarm states” like in my “Electro Mechanical” Time Switch experiment. However this might be no option for an existing project. Another reason are projects that will somehow use DCF77 for calibration purposes and rely on the knowledge of signal loss to switch to their internal clock.

On the other hand there may be projects that use just “naive sync with DCF77”. For those synthesizing the time signal even in “free” state would be an improvement.

Since I can not know what a project really would need my new super filter will provide just all options. It provides three different outputs.

    1. Filtered Output – This is the only option of the original Super Filter – If the clock is at least in state “locked” the output is synthesized and thus 100% clean. The phase will be spot on but it might leap for 1s at unexpected times if can not properly decode the leap second anouncement bit while a leap second is actually scheduled. In less than “locked” states it will pass the signal through. This is the most defensive modes which provides least filtering. If your DCF77 signal is not total crap it may be more than sufficient.
    2. Semi Synthesized Output – This is almost the same as above. However it keeps synthesizing in “unlocked” mode. That is it may allow the phase of the second ticks to drift up to 0.3 s before it starts to pass the signal just through. It is a middle ground. I have no clue if someone will need it but it was almost no additional effort to implement this.
    3. Synthesized Output – This is the most agressive filter mode. After its initial sync it will synthesize no matter what the internal state is. This is probably what some of my readers had in mind when they read about the super filter. However beware: if you do not evaluate the diff pin you will never know that it may be running “free” and thus drifting out of sync slowly. Of course if the signal is reasonable every once in a while it will resync though.

If you look into the code you will notice that it is almost the same as before. Only the pin setup and output handling basically trippled. There are also options to disable some of the output modes. I strongly recommend to disable the undesired modes because unless the signal is really really poor the behaviour of the modes is exactly the same.

//
//  www.blinkenlight.net
//
//  Copyright 2015 Udo Klein
//
//  This program is free software: you can redistribute it and/or modify
//  it under the terms of the GNU General Public License as published by
//  the Free Software Foundation, either version 3 of the License, or
//  (at your option) any later version.
//
//  This program is distributed in the hope that it will be useful,
//  but WITHOUT ANY WARRANTY; without even the implied warranty of
//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
//  GNU General Public License for more details.
//
//  You should have received a copy of the GNU General Public License
//  along with this program. If not, see http://www.gnu.org/licenses/

#include <dcf77.h>


const uint8_t dcf77_analog_sample_pin = 5;
const uint8_t dcf77_sample_pin = 19; // A5
const uint8_t dcf77_inverted_samples = 1;
const uint8_t dcf77_analog_samples = 1;

const uint8_t dcf77_monitor_pin = 18; // A4

const bool provide_filtered_output = true;
const uint8_t dcf77_filtered_pin          = 12;
const uint8_t dcf77_inverted_filtered_pin = 11;
const uint8_t dcf77_filter_diff_pin       = 7;

const bool provide_semi_synthesized_output = true;
const uint8_t dcf77_semi_synthesized_pin          = 16;
const uint8_t dcf77_inverted_semi_synthesized_pin = 15;
const uint8_t dcf77_semi_synthesized_diff_pin     = 14;

const bool provide_synthesized_output = true;
const uint8_t dcf77_synthesized_pin          = 6;
const uint8_t dcf77_inverted_synthesized_pin = 5;
const uint8_t dcf77_synthesized_diff_pin     = 4;

const uint8_t dcf77_second_pulse_pin = 10;


const uint8_t dcf77_signal_good_indicator_pin = 3;

volatile uint16_t ms_counter = 0;
volatile DCF77::tick_t tick = DCF77::undefined;


template <bool enable, uint8_t threshold,
          uint8_t filtered_pin, uint8_t inverted_filtered_pin, uint8_t diff_pin>
void set_output(uint8_t clock_state, uint8_t sampled_data, uint8_t synthesized_signal)
{
    if (enable) {
        const uint8_t filtered_output = clock_state < threshold? sampled_data: synthesized_signal;
        digitalWrite(filtered_pin, filtered_output);
        digitalWrite(inverted_filtered_pin, !filtered_output);
        digitalWrite(diff_pin, filtered_output ^ sampled_data);
    }
}

uint8_t sample_input_pin() {
    const uint8_t clock_state = DCF77_Clock::get_clock_state();
    const uint8_t sampled_data =
        dcf77_inverted_samples ^ (dcf77_analog_samples? (analogRead(dcf77_analog_sample_pin) > 200)
                                                      : digitalRead(dcf77_sample_pin));

    digitalWrite(dcf77_monitor_pin, sampled_data);
    digitalWrite(dcf77_second_pulse_pin, ms_counter < 500 && clock_state>=DCF77_Clock::locked);

    const uint8_t synthesized_signal =
        tick == DCF77::long_tick  ? ms_counter < 200:
        tick == DCF77::short_tick ? ms_counter < 100:
        tick == DCF77::sync_mark  ?  0:
                                           // tick == DCF77::undefined --> default handling
                                           // allow signal to pass for the first 200ms of each second
                                          (ms_counter <=200 && sampled_data) ||
                                           // if the clock has valid time data then undefined ticks
                                           // are data bits --> first 100ms of signal must be high
                                           ms_counter <100;

    set_output<provide_filtered_output, DCF77_Clock::locked,
              dcf77_filtered_pin, dcf77_inverted_filtered_pin, dcf77_filter_diff_pin>
              (clock_state, sampled_data, synthesized_signal);

    set_output<provide_semi_synthesized_output, DCF77_Clock::unlocked,
               dcf77_semi_synthesized_pin, dcf77_inverted_semi_synthesized_pin, dcf77_semi_synthesized_diff_pin>
               (clock_state, sampled_data, synthesized_signal);

    set_output<provide_synthesized_output, DCF77_Clock::free,
               dcf77_synthesized_pin, dcf77_inverted_synthesized_pin,  dcf77_synthesized_diff_pin>
               (clock_state, sampled_data, synthesized_signal);


    ms_counter+= (ms_counter < 1000);

    return sampled_data;
}


void output_handler(const DCF77_Clock::time_t &decoded_time) {
    // reset ms_counter for 1 Hz ticks
    ms_counter = 0;

    // status indicator --> always on if signal is good
    //                      blink 3s on 1s off if signal is poor
    //                      blink 1s on 3s off if signal is very poor
    //                      always off if signal is bad
    const uint8_t clock_state = DCF77_Clock::get_clock_state();
    digitalWrite(dcf77_signal_good_indicator_pin,
                 clock_state >= DCF77_Clock::locked  ? 1:
                 clock_state == DCF77_Clock::unlocked? (decoded_time.second.digit.lo & 0x03) != 0:
                 clock_state == DCF77_Clock::free    ? (decoded_time.second.digit.lo & 0x03) == 0:
                                                       0);
    // compute output for signal synthesis
    DCF77::time_data_t now;
    now.second                    = BCD::bcd_to_int(decoded_time.second);
    now.minute                    = decoded_time.minute;
    now.hour                      = decoded_time.hour;
    now.weekday                   = decoded_time.weekday;
    now.day                       = decoded_time.day;
    now.month                     = decoded_time.month;
    now.year                      = decoded_time.year;
    now.uses_summertime           = decoded_time.uses_summertime;
    now.leap_second_scheduled     = decoded_time.leap_second_scheduled;
    now.timezone_change_scheduled = decoded_time.timezone_change_scheduled;

    now.undefined_minute_output                         = false;
    now.undefined_uses_summertime_output                = false;
    now.undefined_abnormal_transmitter_operation_output = false;
    now.undefined_timezone_change_scheduled_output      = false;

    DCF77_Encoder::advance_minute(now);

    tick = DCF77_Encoder::get_current_signal(now);
}

void setup_serial() {
    Serial.begin(115200);
}

void output_splash_screen() {
    using namespace DCF77_Encoder;

    Serial.println();
    Serial.println(F("DCF77 Superfilter 1.1.0"));
    Serial.println(F("(c) 2015 Udo Klein"));
    Serial.println(F("www.blinkenlight.net"));
    Serial.println();
    Serial.print(F("Sample Pin:               ")); Serial.println(dcf77_sample_pin);
    Serial.print(F("Inverted Mode:            ")); Serial.println(dcf77_inverted_samples);
    Serial.print(F("Analog Mode:              ")); Serial.println(dcf77_analog_samples);
    Serial.print(F("Monitor Pin:              ")); Serial.println(dcf77_monitor_pin);
    Serial.println();

    if (provide_filtered_output) {
        Serial.println(F("Filtered Output"));
        Serial.print(F("    Filtered Pin:         ")); Serial.println(dcf77_filtered_pin);
        Serial.print(F("    Diff Pin:             ")); Serial.println(dcf77_filter_diff_pin);
        Serial.print(F("    Inverse Filtered Pin: ")); Serial.println(dcf77_inverted_filtered_pin);
        Serial.println();
    }

    if (provide_semi_synthesized_output) {
        Serial.println(F("Semi Synthesized Output"));
        Serial.print(F("    Filtered Pin:         ")); Serial.println(dcf77_semi_synthesized_pin);
        Serial.print(F("    Diff Pin:             ")); Serial.println(dcf77_semi_synthesized_diff_pin);
        Serial.print(F("    Inverse Filtered Pin: ")); Serial.println(dcf77_inverted_semi_synthesized_pin);
        Serial.println();
    }

    if (provide_synthesized_output) {
        Serial.println(F("Synthesized Output"));
        Serial.print(F("    Filtered Pin:         ")); Serial.println(dcf77_synthesized_pin);
        Serial.print(F("    Diff Pin:             ")); Serial.println(dcf77_synthesized_diff_pin);
        Serial.print(F("    Inverse Filtered Pin: ")); Serial.println(dcf77_inverted_synthesized_pin);
        Serial.println();
    }

    Serial.print(F("Second Pulse Pin:         ")); Serial.println(dcf77_second_pulse_pin);
    Serial.print(F("Signal Good Pin:          ")); Serial.println(dcf77_signal_good_indicator_pin);

    Serial.println();

    Serial.println();
    Serial.println(F("Initializing..."));
    Serial.println();
};

void setup_pins() {
    if (provide_filtered_output) {
        pinMode(dcf77_filtered_pin, OUTPUT);
        pinMode(dcf77_filter_diff_pin, OUTPUT);
        pinMode(dcf77_inverted_filtered_pin, OUTPUT);
    }

    if (provide_semi_synthesized_output) {
        pinMode(dcf77_semi_synthesized_pin, OUTPUT);
        pinMode(dcf77_semi_synthesized_diff_pin, OUTPUT);
        pinMode(dcf77_inverted_semi_synthesized_pin, OUTPUT);
    }

    if (provide_synthesized_output) {
        pinMode(dcf77_synthesized_pin, OUTPUT);
        pinMode(dcf77_synthesized_diff_pin, OUTPUT);
        pinMode(dcf77_inverted_synthesized_pin, OUTPUT);
    }

    pinMode(dcf77_monitor_pin, OUTPUT);
    pinMode(dcf77_signal_good_indicator_pin, OUTPUT);
    pinMode(dcf77_second_pulse_pin, OUTPUT);
    pinMode(dcf77_sample_pin, INPUT);
    digitalWrite(dcf77_sample_pin, HIGH);
}



void setup_clock() {
    DCF77_Clock::setup();
    DCF77_Clock::set_input_provider(sample_input_pin);
    DCF77_Clock::set_output_handler(output_handler);
}

void setup() {
    setup_serial();
    output_splash_screen();
    setup_pins();
    setup_clock();
}

void loop() {
    DCF77_Clock::time_t now;

    DCF77_Clock::get_current_time(now);

    if (now.month.val > 0) {
        Serial.println();
        Serial.print(F("Decoded time: "));

        DCF77_Clock::print(now);
        Serial.println();
    }

    DCF77_Clock::debug();
}

If you look at the code there is almost nothing new compared to the old super filter. Of course the output handling basically trippled as it now provides 3 outputs. The only noteworthy thing is the template function set_output. Notice that the if statement is NOT evaluated at run time but at compile time. As a consequence the compiler will optimize the function away for the modes that are not in use.

8 Responses to Super Filter and Synthesizer

  1. Ralph Drewello says:

    Hi Udo, thank you very much – I’ve successfully tested your awesome Superfilter with my own clock project based on Atmega 644/1284. Now I am thinking about how to get the status information out of the Superfilter into the “main” processor (perhaps to display signal quality on grafic display). Sure I could read the status line (LED) with a free port but I don’t think that this would be the best way. What do you think about a serial connection which could deliver more information from the Superfilter with some basic commands?

    During repeated tests with (really) poor DCF-77 signal (blink 1s-on/3s-off) I’ve noticed a wrong time after sync (was done after a couple of hours) with main processor using the “synthesized” output of the Superfilter (which is running on an Arduino Nano with 16MHz Quartz). Any Idea?

    Now I am using the “filtered” output because I deployed an RTC (DS3232) on the main board to preserve precise time even DCF signal is poor or lost and to have the valid time immediately after startup or power loss.

    Best regards
    Ralph

    • Hi Ralph, in order to analyze this I need more information. Which version of my library are you running? In order to get an overall idea of your signal situation. Could you please provide me a log of the received signal logged by the “Swiss Army Debug Helper” sketch in Mode “Dm” for at least an hour. Then I would need the log output of the super filter running long enough till it reaches a state where you problem occurs. These two logs should allow me to infer what is going on.

      Of course you can extend the superfilter with more information. However why don’t you then just ditch the superfilter and use the library directly in the main CPU? If you are afraid if you have sufficient processing power just switch to Arduino Due / ARM processor. This has way more computational power than two ATmegas combined and overall system complexity would go down.

  2. Ralph Drewello says:

    Hi Udo, thanks a lot for your fast reply. While setting up some test I’ve got some strange results – using Arduino IDE 1.8.2 and your latest Library version 3.1.8. First I’ve noticed some Compiler warnings but can’t judge if this is really important or not. After uploading the code to the board it seams ok for some minutes. The DCF-Signal was good (for this test). So I’ve got the status from the debugger with the current (valid) time 00:02:58.

    0 ??.05.03(3,?)00:02:57 MEZ 0,0 0 p(2958-0:184) s(23-2:5) m(10-6:1) h(8-4:1) wd(6-4:0) D(8-6:0) M(8-6:0) Y(8-7:0) 4,2,2,255
    [1] 191, +———+——–9XXXXXXXXX3+———+———+———+———+———+———+———
    [2] 191, +———+———+———+———+———+———+———+———+———+———
    0 17.05.03(3,3)00:02:58 MEZ 0,0 0 p(2970-0:185) s(23-2:5) m(10-6:1) h(8-4:1) wd(6-4:0) D(8-6:0) M(8-6:0) Y(8-6:0) 4,2,2,255
    [1] 192, +———+——-5XXXXXXXXXX6+———+———+———+———+———+———+———
    [2] 192, +———+———+———+———+———+———+———+———+———+———

    However some seconds later the disappeared:

    0 17.05.03(?,3)00:03:57 MEZ 0,0 0 p(3507-0:219) s(31-2:5) m(14-12:0) h(12-6:1) wd(7-6:0) D(12-9:0) M(12-9:0) Y(12-10:0) 6,3,1,255
    [1] 251, +———+———+8XXXXXXXXXX6——-+———+———+———+———+———+———
    [2] 251, +———+———+———+———+———+———+———+———+———+———
    0 ??.05.03(?,?)00:03:58 MEZ 0,0 0 p(3515-0:219) s(31-2:5) m(14-12:0) h(12-6:1) wd(7-6:0) D(12-9:0) M(12-9:0) Y(12-11:0) 6,3,1,255
    [1] 252, +———+———+8XXXXXXXXX6——–+———+———+———+———+———+———
    [2] 252, +———+———+———+———+———+———+———+———+———+———

    and later even the minute

    0 ??.05.03(3,?)00:05:28 MEZ 0,0 0 p(4023-0:251) s(41-7:6) m(18-16:0) h(8-4:1) wd(10-8:0) D(12-10:0) M(14-10:1) Y(14-14:0) 8,3,1,255
    [1] 342, +———+———+—-8XXXXXXXXXXX—+———+———+———+———+———+———
    [2] 342, +———+———+———+———+———+———+———+———+———+———
    0 ??.05.03(3,?)00:??:29 MEZ 0,0 0 p(4030-0:251) s(41-7:6) m(20-20:0) h(8-4:1) wd(10-8:0) D(12-10:0) M(14-10:1) Y(14-14:0) 8,3,1,255
    [1] 343, +———+———+—-9XXXXXXXXXX8—+———+———+———+———+———+———
    [2] 343, +———+———+———+———+———+———+———+———+———+———

    and so on … and then here are the wrong numbers …

    0 77.07.??(7,?)??:??:19 MEZ 0,0 0 p(4139-28:255) s(75-0:12) m(36-36:0) h(60-60:0) wd(42-27:2) D(57-57:0) M(42-29:2) Y(63-52:2) 8,14,18,255
    [1] 1353, +———+———+———+———+———+———+——–5XXXXXXXXXX2———+———
    [2] 1353, +———+———+———+———+———+———+———+———+———+———

    Now I am not sure if I’ve got wrong. I will restart the test now and will see in the morning which status will achieved.

    BTW I’ve tried the IDE 1.8.0 and 1.6.7 with the similar results. In the beginning I’ve used the library Version 3.0 with IDE 1.6.7 to avoid the Preprocessor Issue.

    Best Regards
    Ralph

    P.S: Simple question – how can I send you the log files?

    • Until you send me a copy of the warnings and tell me which settings (Host PC Operating System, target board) I can not judge either. With regard to the logs: email or github are both fine.

      • Ralph Drewello says:

        I could identify that the problem came from the Arduino Nano HW Modul itself (imported directly from China). Trying another HW Module while using the same Library version (3.1.8) and SW environment (IDE 1.8.2 running on Win7/64bit) I could get a running Filter without any problems now – works very stable even if the DCF signal is noisy.

        The Compiler warnings are regarding the usage of possibly uninitialized variables – probaply not relevant – are seen with Compiler SW “show warning = all”:

        dcf77.cpp:1639:69: warning: ‘precision’ may be used uninitialized in this function [-Wmaybe-uninitialized] .. eeprom_write_byte((uint8_t *)(eeprom++), (uint8_t) precision);

        dcf77.cpp:1688:49: warning: ‘adjust’ may be used uninitialized in this function [-Wmaybe-uninitialized] .. ( confirmed_precision == 1 &&

        Regarding ARM processor I will evaluate if your Library and my project will work together on this platform (Arduino Due already ordered) soon. Of course this will be a new challenge but the extended capabilities and the additional memory!! are worth. If this will be successfull I will develop a new PCB for such a wall mounted LED Clock as I already have with a lot of additional features (e.g. sensors, remote control, sound) and an industrially manufactured aluminum casing. If you are interested I would discuss the project if I am ready with the new design.

        Last but not least I will continue the testing of the superfilter (blackbox) on my current clock design. Again thank you very much.

        Best regards
        Ralph

      • You may ignore the warnings. The values are either properly initialized or it is the task of the function to fill the them. In both cases the warning does not apply.

  3. Ralph Drewello says:

    Hi Udo, this morning I’ve tried another Arduino (Nano) module which was programmed last week (Library Version 3.1.4) and this is working ok. See the output below:

    0 17.05.03(3,3)07:10:57 MEZ 0,0 0 p(6125-0:255) s(39-2:7) m(20-12:2) h(24-16:2) wd(12-8:1) D(16-12:1) M(16-12:1) Y(16-13:0) 8,4,4,255
    [1] 316, +———+——-9XXXXXXXXX6-+———+———+———+———+———+———+———
    [2] 316, +———+———+———+———+———+———+———+———+———+———

    Decoded time: 17-05-03 3 07:09:59 CEST ..
    5 17.05.03(3,3)07:10:58 MESZ 0,0 2 p(6144-0:255) s(39-2:7) m(20-12:2) h(24-16:2) wd(12-8:1) D(16-12:1) M(16-12:1) Y(16-12:1) 8,4,4,255
    [1] 317, +———+——5XXXXXXXXXXX3+———+———+———+———+———+———+———
    [2] 317, +———+———+———+———+———+———+———+———+———+———

    Signal good LED is on – so with this setup I will now try to repeat the issue with the poor DCF signal which results in a wrong time after a couple of hours.

    I am wondering why its was not possible to program a new Modul with the latest SW Version yesterday. Perhaps I did a mistake or there is something wrong with the SW provision. I’ve used the Library Manager to install the Software on a WIndos 7 (64bit) System in both ways (add .zip Library with manually download from Github and Manage Library). Same result as described.

    BTW: I will check my project on an ARM processor although I’ve already have a PCB designed for Atmega. Because I need to add the superfilter I am thinking about a redesign and therefore its a good time to switch to another (more powerful) processor. Thanks a lot.

    • The library manager pulls the most current version from Github. It should never be more than 1 hour behind. So this makes no difference. But what do you mean by “I was not able to … ?” What exactly failed and how? With regard to the ARM processor: a lot of people claim that the ARMs are “harder to use” compared to the Atmega. From my experience this is not true. If you restrict yourself to the reduced featureset offered by the Atmegas I do not find them harder to use. However the ARM is more capable and if you leverage all features, then it is surely more challenging. On the other hand: if you need leveled interrupts or more memory and have only an ATmega you are basically screwed. With other words: if you are OK with 3.3V and do not have a very tight power budget then ARM is often a better choice. Also my measurements indicate that the ARM based version of my clock will have even better noise tolerance under marginal reception conditions. This is because I can leverage more memory for the computation of the phase lock.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s