Jose Vu Blog

random

Tech stack I’m using

By josevu

1. Programming Languages

F#

  • Use Case: Powers the majority of automation flows and back-end services.
  • Pros:
    • Functional-first approach leads to concise, maintainable code.
    • Strong type system reduces bugs and streamlines refactoring.
    • Backed by Microsoft, with excellent tooling in Visual Studio/VS Code.
    • Asynchronous workflows are straightforward for background processing.
    • .NET ecosystem enables easier debugging and testing.

Python

  • Use Case: Data science workflows, automation scripts, and prototyping.
  • Pros:
    • Massive library ecosystem for data manipulation and machine learning.
    • Key Libraries:
      • pandas for flexible data manipulation and quick exploratory analysis.
      • scipy for advanced scientific computing (e.g., optimization, signal processing).
      • statsmodels for statistical modeling and time-series analysis.
    • Widely adopted, making it easier to hire and collaborate.
    • Integrates seamlessly with Jupyter notebooks for interactive development.

SQL

  • Use Case: Core query language for relational databases and data transformations.
  • Pros:
    • Industry-standard for structured data handling.
    • Excellent for aggregations, joins, and analytics.
    • Compatible with a vast range of tooling and platforms.

PHP

  • Use Case: Maintaining older or legacy web applications.
  • Pros:
    • Large, established ecosystem.
    • Straightforward approach for minor enhancements in existing codebases.

2. Database & Storage

TimescaleDB (on PostgreSQL)

  • Use Case: Primary data repository, optimized for time-series workloads.
  • Pros:
    • Built on PostgreSQL, a mature open-source database with broad tooling support.
    • Time-series extensions yield efficient historical and real-time queries.
    • ACID transactions, powerful indexing, and comprehensive feature set.

SQLite

  • Use Case: Lightweight, file-based relational database for local development or embedded applications.
  • Pros:
    • Serverless and zero-configuration, ideal for quick stand-ups and prototypes.
    • Fast, reliable, and easily embedded in various environments.

3. Data Visualization & Monitoring

Grafana

  • Use Case: Real-time monitoring, metrics dashboards, and alerting.
  • Pros:
    • Flexible for ingesting data from multiple sources.
    • Rich plugin ecosystem and strong community support.
    • Straightforward dashboards for both technical and non-technical users.

Holistics

  • Use Case: Modern business intelligence (BI) with SQL-based modeling.
  • Pros:
    • Easy to adopt if your team is already SQL-savvy.
    • Lightweight solution for interactive dashboards and data exploration.

Looker

  • Use Case: Enterprise BI and data modeling via LookML.
  • Pros:
    • Powerful semantic layer with centralized data definitions.
    • Scales well for organizations with complex data modeling needs.

Tableau

  • Use Case: Visual analytics and interactive dashboard creation.
  • Pros:
    • Intuitive drag-and-drop interface for quick dashboard building.
    • Strong community with a wealth of tutorials and extensions.

SAS JMP

  • Use Case: Statistical analysis and interactive data exploration.
  • Pros:
    • Specialized tools for advanced analytics and design of experiments (DOE).
    • Clear, visual approach to exploring data relationships.

Metabase

  • Use Case: Lightweight self-service analytics, ideal for quick insights.
  • Pros:
    • Very user-friendly, particularly for small teams or less technical users.
    • Minimal overhead; fast to deploy and learn.

4. Source Code Version Control

GitHub

  • Use Case: Central hub for all codebases (F#, Python, SQL, etc.).
  • Pros:
    • Industry-standard for distributed version control (Git).
    • Pull requests, issues, and CI/CD integrations built in.
    • Wide range of third-party apps (e.g., container registries, project management tools).

5. Integration & Deployment

Docker (Podman in consideration)

  • Use Case: Packaging and deploying applications in consistent, portable containers.
  • Pros:
    • Simplifies environment setups and dependency management.
    • Enables easy scaling, rolling updates, and rollbacks.
    • Podman offers a daemonless architecture for enhanced security.

6. Automation Orchestrator

Portainer

  • Use Case: GUI-based management for Docker containers, images, and networks.
  • Pros:
    • Simplifies daily container operations.
    • Visual interface for monitoring container health and resource usage.
    • Minimizes command-line overhead.

Cronicle

  • Use Case: Scheduling and orchestrating automated jobs or tasks.
  • Pros:
    • Web-based UI for setting up schedules and monitoring job history.
    • Suitable for simple cron-like tasks and more complex workflows.
    • Centralized logging and alerting helps prevent overlooked failures.

Metaflow

  • Use Case: Data science workflow orchestration and pipeline management.
  • Pros:
    • Pythonic API, making it accessible to data scientists.
    • Built-in versioning, retries, and metadata tracking for reproducible pipelines.
    • Good for end-to-end workflows, from data extraction to model deployment.

7. Data Transformation

dbt (Data Build Tool)

  • Use Case: Transforming and modeling data in a warehouse context (e.g., PostgreSQL, TimescaleDB).
  • Pros:
    • Modular, version-controlled SQL transformations.
    • Built-in testing and documentation for higher data quality.
    • Strong open-source community and plugin ecosystem.
    • Integrates seamlessly with GitHub for CI/CD.

8. Observability

OpenTelemetry

  • Use Case: Collecting logs, metrics, and traces in a standardized format.
  • Pros:
    • Vendor-neutral standard, reducing lock-in to proprietary tools.
    • Wide language coverage, from .NET to Python and more.
    • Integrates easily with existing monitoring setups (Grafana, Jaeger, etc.).
    • Helps achieve end-to-end visibility in distributed systems.

Key Takeaways

  1. F# for Automation, Python for Analytics: A powerful duo covering functional back-end work and rich data science ecosystems.
  2. Robust Databases: TimescaleDB/PostgreSQL for scalable time-series data; SQLite for lightweight or embedded scenarios.
  3. Multiple Visualization Tools: From real-time monitoring (Grafana) to BI solutions (Holistics, Looker, Tableau, SAS JMP) for in-depth analysis.
  4. Version Control & Deployment: GitHub pairs with Docker/Podman to streamline CI/CD and container-based deployment.
  5. Automation & Workflow Management: Portainer for container management, Cronicle for scheduling, and Metaflow for data science pipelines.
  6. Structured Data Transformation: dbt enforces consistency and best practices for SQL modeling.
  7. Holistic Observability: OpenTelemetry ensures standardized logging, metrics, and tracing across services.

By combining these tools and services, you can build a scalable, maintainable, and efficient environment that addresses everything from quick prototyping to production-level analytics and automated workflows.

References

By josevu

Related Post

random

Hello world!

josevu

Leave a Reply

Your email address will not be published. Required fields are marked *