My Year in Review (2020)
Last year, I thought a few things would be coming in 2020.
What's Coming in 2020
- Cloud Enterprise Data Platforms
- Hybrid Cloud
- Streaming with Flink, Kafka, NiFi
- AI at the Edge with Microcontrollers and Small Devices
- Voice Data In Queries
- Event Handler as a Service (Automatic Kafka Message Reading)
- More Powerful Parameter Based Modular Streaming
- Cloud First For Big Data
- Log Handling Moves to MiNiFi
- Full AI At The Edge with Deployable Models
- More Powerful Edge TPU/GPU/VPU
- Kafka is everywhere
- Open Source UI Driven Event Engines
- FLaNK Stack gains popularity
- FLINK Everywhere
Some of this was deferred due to the global pandemic and I had a big miss with Voice Data and some advances in streaming getting delayed.
Here's a list bit of new news for 2020, SRM was added to Kafka DataHub in the Public Cloud:
https://docs.cloudera.com/runtime/7.2.6/srm-overview/topics/srm-replication-overview.html
I also just did a best of 2020 video you can check out:
Articles from 2020
- https://dzone.com/articles/edgeai-google-coral-with-coral-environmental-senso
- https://dzone.com/articles/lets-build-a-simple-ingest-to-cloud-data-warehouse
- https://dzone.com/articles/real-time-streaming-deep-learning-pipelines-with-d
- https://community.cloudera.com/t5/Community-Articles/nbsp-FLaNK-Smart-Weather-Applications-with-Flink-SQL-nbsp/ta-p/305905
- https://www.datainmotion.dev/2020/12/cloudera-data-platform-using-apache.html
- https://www.datainmotion.dev/2020/12/basic-understanding-of-cloudera-flow.html
- https://www.datainmotion.dev/2020/11/building-ssl-for-hosting-mobile-sites.html
- https://www.datainmotion.dev/2020/12/new-release-for-hdf-352-and-cloudera.html
- https://www.datainmotion.dev/2020/11/flank-smart-weather-websocket.html
- https://www.datainmotion.dev/2020/11/flank-smart-weather-applications-with.html
- https://www.datainmotion.dev/2020/10/flank-streaming-edgeai-on-new-nvidia.html
- https://www.datainmotion.dev/2020/10/2020-events-slides-githubs-and-videos.html
- https://www.datainmotion.dev/2020/10/top-25-use-cases-of-cloudera-flow.html
- https://www.datainmotion.dev/2020/10/running-flink-sql-against-kafka-using.html
- https://www.datainmotion.dev/2020/10/automating-building-migration-backup.html
- https://www.datainmotion.dev/2020/10/monitoring-mac-laptops-with-apache-nifi.html
- https://www.datainmotion.dev/2020/10/tracking-satellites-with-apache-nifi.html
- https://www.datainmotion.dev/2020/10/unveiling-nvidia-jetson-nano-2gb-and.html
- https://www.datainmotion.dev/2020/09/devops-working-with-parameter-contexts.html
- https://www.datainmotion.dev/2020/09/using-google-forms-as-a-data-source-for.html
- https://www.datainmotion.dev/2020/09/using-djlai-for-deep-learning-bert-q-in.html
- https://www.datainmotion.dev/2020/09/using-djlai-for-deep-learning-based.html
- https://www.datainmotion.dev/2020/09/cloudera-edge-flow-updates-cloudera.html
- https://www.datainmotion.dev/2020/09/cloudera-streams-messaging-manager.html
- https://www.datainmotion.dev/2020/09/apache-flink-rest-and-metrics.html
- https://www.datainmotion.dev/2020/09/data-and-streaming-news-september-2020.html
- https://www.datainmotion.dev/2020/08/apache-nifi-112-released-18-august-2020.html
- https://www.datainmotion.dev/2020/08/deleting-schemas-from-cloudera-schema.html
- https://www.datainmotion.dev/2020/08/did-user-really-ask-for-exactly-once.html
- https://www.datainmotion.dev/2020/07/flank-in-cloud-huge-cloudera-data.html
- https://www.datainmotion.dev/2020/07/phoenix-hbase-nifi-resources.html
- https://www.datainmotion.dev/2020/07/sizing-your-apache-nifi-cluster-for.html
- https://www.datainmotion.dev/2020/07/report-on-this-apache-nifi-1114-monitor.html
- https://www.datainmotion.dev/2020/07/ingesting-all-weather-data-with-apache.html
- https://www.datainmotion.dev/2020/07/apache-flink-sql-demo-flank-series.html
- https://www.datainmotion.dev/2020/07/using-cloudera-data-platform-with-flow.html
- https://www.datainmotion.dev/2020/06/the-rise-of-mega-edge-flank.html
- https://www.datainmotion.dev/2020/06/explore-enterprise-apache-flink-with.html
- https://www.datainmotion.dev/2020/06/using-apache-kafka-using-cloudera-data.html
- https://www.datainmotion.dev/2020/06/unboxing-most-amazing-edge-ai-device.html
- https://www.datainmotion.dev/2020/06/discord-integration-with-apache-nifi.html
- https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html
- https://www.datainmotion.dev/2020/05/commonly-used-tcpip-ports-in-streaming.html
- https://www.datainmotion.dev/2019/11/cloudera-edge-management-110-release.html
- https://www.datainmotion.dev/2020/05/lets-query-kafka-with-hive.html
- https://www.datainmotion.dev/2020/05/cloudera-flow-management-101-lets-build.html
- https://www.datainmotion.dev/2020/05/one-minute-nifi-tip-calcite-sql-notes.html
- https://www.datainmotion.dev/2020/05/flank-low-code-streaming-populating.html
- https://www.datainmotion.dev/2020/05/available-now-cfm-110.html
- https://www.datainmotion.dev/2020/05/time-series-analysis-dataflow.html
- https://www.datainmotion.dev/2020/05/flink-sql-preview.html
- https://www.datainmotion.dev/2020/04/streaming-data-with-cloudera-data-flow.html
- https://www.datainmotion.dev/2020/04/apache-nifi-and-nifi-registry.html
- https://www.datainmotion.dev/2020/04/building-search-indexes-with-apache.html
- https://www.datainmotion.dev/2020/04/harnessing-data-lifecycle-for-customer.html
- https://www.datainmotion.dev/2020/04/predicting-sensor-readings-with-time.html
- https://www.datainmotion.dev/2020/04/sql-reporting-task-for-cloudera-flow.html
- https://www.datainmotion.dev/2020/04/read-apache-impala-apache-kudu-tables.html
- https://www.datainmotion.dev/2020/03/using-nifi-cli-to-restore-nifi-flows.html
- https://www.datainmotion.dev/2020/03/fixing-linux-webcams.html
- https://www.datainmotion.dev/2020/03/odpis-opends4all-open-source-data.html
- https://www.datainmotion.dev/2020/02/new-and-improved-its-nifi.html
- https://www.datainmotion.dev/2020/02/edgeai-google-coral-with-coral.html
- https://www.datainmotion.dev/2020/02/connecting-apache-nifi-to-apache-atlas.html
- https://www.datainmotion.dev/2020/02/example-smm-notification-email.html
- https://www.datainmotion.dev/2020/02/apache-atlas-for-monitoring-edge2ai-iot.html
- https://www.datainmotion.dev/2020/02/edgeai-jetson-nano-with-minifi-c-agent.html
- https://www.datainmotion.dev/2020/01/quick-tip-nifi-json-cleanup.html
- https://www.datainmotion.dev/2020/01/quicktip-ingesting-google-analytics-api.html
- https://www.datainmotion.dev/2020/01/analyzing-wood-burning-stoves-with_23.html
- https://www.datainmotion.dev/2020/01/analyzing-wood-burning-stoves-with.html
- https://www.datainmotion.dev/2020/01/flank-stack-nifi-processor-for-kafka.html
- https://www.datainmotion.dev/2020/01/cloudera-edge2ai-minifi-java-agent-with.html
- https://www.datainmotion.dev/2020/01/year-and-decade-end-report-201.html
- https://www.datainmotion.dev/2020/12/smart-stocks-with-flank-nifi-kafka.html
- https://www.datainmotion.dev/2020/12/simple-change-data-capture-cdc-with-sql.html
- https://www.datainmotion.dev/2020/12/ingesting-websocket-data-for-live-stock.html
Talks Around the World (Virtual)
June 11, 2020 - INRHYTHM Lightning Talk
August 13, 2020 - Real-Time Analytics
https://www.cloudera.com/about/events/virtual-events/realworld-streaming-analytics.html
August 28, 2020 - Apache Beam Summit
September 10, 2020 - Future of Data Princeton Meetup - Flink Flank Flink Fest
September 29 - October 1, 2020 - ApacheCon
September 29, 2020 - Apache Streaming Meetup
October 9, 2020 - DevOps Stage
October 22, 2020 - Flink Forward
October 26, 2020 - Open Source Summit Europe
October 27, 2020 - AI Dev World
November 10, 2020 - Variis Talk
November 25, 2020 - Big Data Conference
December 14, 2020 - Apache MXNet Day
https://www.eventbrite.com/e/apache-mxnet-day-dec-14th-900-to-500-pm-pst-tickets-127767842055#
Source Code
- https://github.com/tspannhw/SmartStocks
- https://github.com/tspannhw/ClouderaFlinkSQLForPartners
- https://github.com/tspannhw/ClouderaPublicCloudCDFWorkshop
- https://github.com/tspannhw/EverythingApacheNiFi
- https://github.com/tspannhw/MXNetDay2020
- https://github.com/tspannhw/SmartWeather
- https://github.com/tspannhw/tspannhw.github.io
- https://github.com/tspannhw/table-ddl
- https://github.com/tspannhw/flink-sql-queries
- https://github.com/tspannhw/webpages
- https://github.com/tspannhw/SettingUpAJetsonNano2GB
- https://github.com/tspannhw/minifi-jetson-nano
- https://github.com/tspannhw/movies-social
- https://github.com/tspannhw/nifi-osquery
- https://github.com/tspannhw/SchemaRegistry
- https://github.com/tspannhw/retail-dynamic-shelf-pricing
- https://github.com/tspannhw/ClouderaFlowManagementWorkshop
- https://github.com/tspannhw/conferences
- https://github.com/tspannhw/minifi-xaviernx
- https://github.com/tspannhw/FlinkForwardGlobal2020
- https://github.com/tspannhw/StreamingDemos
- https://github.com/tspannhw/nifi-djlqa-processor
- https://github.com/tspannhw/FlinkSQLWithCatalogsDemo
- https://github.com/tspannhw/nifi-djl-processor
- https://github.com/tspannhw/nifi-djlsentimentanalysis-processor
- https://github.com/tspannhw/ApacheDeepLearning301
- https://github.com/tspannhw/CloudDataFlowWorkshop
- https://github.com/tspannhw/DevOpsStage2020
- https://github.com/tspannhw/minifi-jetbot
- https://github.com/tspannhw/tutorial
- https://github.com/tspannhw/flume-to-nifi
- https://github.com/tspannhw/tspannhw
- https://github.com/tspannhw/meetup-sensors
- https://github.com/tspannhw/FlinkSQLDemo
- https://github.com/tspannhw/ClouderaStreamingAnalyticsSQLWorkshop
- https://github.com/tspannhw/cdp-datahub-azure-nifikafka
- https://github.com/tspannhw/ClouderaNow2020
- https://github.com/tspannhw/airline-sentiment-streaming
- https://github.com/tspannhw/EmergingTechnologyDay2020
- https://github.com/tspannhw/NiFItoKafkaConnect
- https://github.com/tspannhw/NiFiToDiscord
- https://github.com/tspannhw/ExecuteClouderaML
- https://github.com/tspannhw/nifi-solr-example
- https://github.com/tspannhw/pyportalhacking
- https://github.com/tspannhw/2020Demos
- https://github.com/tspannhw/flows
- https://github.com/tspannhw/minifi-gasthermal
- https://github.com/tspannhw/kafkarest-processor
- https://github.com/tspannhw/MmFLaNK
- https://github.com/tspannhw/ApacheConAtHome2020
Slides From Events and Talks 2020
- https://www2.slideshare.net/bunkertor/using-the-flank-stack-for-edge-ai-apache-mxnet-apache-flink-apache-nifi-apache-kafka-apache-kudu-lightning?ref=
- https://www2.slideshare.net/bunkertor/using-apache-mx-net-in-production-deep-learning-streaming-pipelines
- https://www2.slideshare.net/bunkertor/fast-data-for-fitness-10-nov-2020
- https://www2.slideshare.net/bunkertor/cloudera-streaming-with-flink-oct-29-2020-meetup-london
- https://www2.slideshare.net/bunkertor/cracking-the-nut-solving-edge-ai-with-apache-tools-and-frameworks
- https://www2.slideshare.net/bunkertor/learning-the-basics-of-apache-nifi-for-iot-oss-europe-2020
- https://www2.slideshare.net/bunkertor/using-the-flank-stack-for-edge-ai-flink-nifi-kafka-kudu
- https://www2.slideshare.net/bunkertor/lets-build-a-simple-ingest-to-cloud-datawarehouse-with-low-code
- https://www2.slideshare.net/bunkertor/real-time-stock-processing-with-apache-nifi-apache-flink-and-apache-kafka
- https://www2.slideshare.net/bunkertor/incrementally-streaming-rdbms-data-to-your-data-lake-automagically
- https://www2.slideshare.net/bunkertor/using-the-flank-stack-for-edge-ai-apache-mxnet-apache-flink-apache-nifi-apache-kafka-apache-kudu
- https://www2.slideshare.net/bunkertor/time-series-analysis-dataflow
- https://www2.slideshare.net/bunkertor/introduction-to-apache-nifi-1114
This Year's Devices
- NVIDIA Jetson Xavier NX
- NVIDIA Jetson Nano 2GB
- Breakout Garden for Raspberry Pi (I2C + SPI)
- SGP30 Air Quality Sensor Breakout
- MLX90640 Thermal Camera Breakout – Wide angle (110°)
- Raspberry Pi 4 - 8G RAM!
What's Coming in 2021
- AI Driven Streaming
- Cloudera Data Platform on GCP
- OpenShift Powered by CDP
- More Hybrid Cloud
- SQL Stream Builder for Flink SQL
- More powerful Edge AI
- More TPUs, GPUs and specialized chips
- More Streaming
- More COVID datasets
- More Robots
- More Automation
- More Serverless
- Voice Data In Queries
- K8 Everywhere...
- K8 Backlash
References:
- https://www.datainmotion.dev/2020/01/year-and-decade-end-report-201.html
- https://community.cloudera.com/t5/Community-Articles/My-Year-in-Review-2018/ta-p/249363
- https://community.cloudera.com/t5/Community-Articles/My-Year-in-Review-2017/ta-p/247541
Top comments (1)
Wow!! You have stuff going on!