Interface ETLSource

  • All Known Implementing Classes:
    MergeDedupStoragePlugin, PlainPBStoragePlugin

    public interface ETLSource
    An ETL source is data source that can act as a source for ETL. There are the methods that an ETL source needs to implement
    • Method Detail

      • getETLStreams

        List<ETLInfo> getETLStreams​(String pv,
                                    Timestamp currentTime,
                                    ETLContext context)
                             throws IOException
        Given a pv and a time, this method returns all the streams that are ready for ETL. For example, if the partition granularity of a source is an hour, then this method returns all the streams that are in this source for the previous hours. Ideally, these streams must be closed for writing and should not change. The ETL process will consolidates these streams into the ETL destination, which is expected to be at a longer time granularity.
        pv - The name of PV.
        currentTime - The time that is being used as the cutoff. If we pass in a timestamp way out into the future, we should return all the streams available.
        context - ETLContext
        List ETLinfo
        IOException -  
      • markForDeletion

        void markForDeletion​(ETLInfo info,
                             ETLContext context)
        Delete the ETLStream identifier by info when you can as it has already been consumed by the ETL destination. You can delete it later or immediately.
        info - ETLInfo
        context - ETLContext
      • getDescription

        String getDescription()
      • consolidateOnShutdown

        boolean consolidateOnShutdown()
        Should ETL move data from this source to the destination on shutdown. For example, if you are using a ramdisk for the STS and you have a UPS, you can minimize any data loss but turning this bit on for data stores that are on the ramdisk. On shutdown, ETL will try to move the data out of this store into the next lifetime store.
        boolean True or False