2 posts tagged with "schema"

Announcing the release of Spice v1.0.7 🔧

Spice v1.0.7 improves memory usage when using DuckDB, improves schema inference performance when using object-store based data connectors, and fixes a bug in Dremio schema inference.

Highlights in v1.0.7​

• DuckDB Memory Usage: Memory usage when using DuckDB has been significantly improved for data loads and refreshes through expanded use of zero-copy Arrow and multi-threading for data loads. When a duckdb_memory_limit is specified, disk spilling has been improved for greater-than-memory workloads. In addition, a new temp_directory runtime parameter supports storing temporary files to alternative location than the DuckDB data file for higher throughput. For example, temp_directory could be set to a different high-IOPs IO2 EBS volume that is separate from the duckdb_file_path.

  Automated end-to-end tests for the DuckDB Accelerator coverage has been significantly expanded.

  For configuration details, see the documentation for runtime parameters and the DuckDB Data Accelerator.

• Schema Inference Performance for Object-Store Data Connectors: Schema inference performance has been improved, especially for large numbers of objects (1M+ objects) when using object-store based data connectors by making the object-listing and selection more efficient.

Contributors​

• @phillipleblanc
• @sgrebnov
• @peasee
• @Sevenannn

Breaking Changes​

No breaking changes.

Upgrading​

To upgrade to v1.0.7, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.0.7 image:

docker pull spiceai/spiceai:1.0.7

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

• fix: Remove on zero results arguments from benchmarks by @peasee in https://github.com/spiceai/spiceai/pull/4533
• Run benchmark tests w/o uploading test results (pending improvements) by @sgrebnov in https://github.com/spiceai/spiceai/pull/4843
• fix: Return BAD_REQUEST when not embeddings are configured by @peasee in https://github.com/spiceai/spiceai/pull/4804
• Fix Dremio schema inference by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5114
• Improve performance of schema inference for object-store data connectors by @sgrebnov in https://github.com/spiceai/spiceai/pull/5124
• Always download spice runtime version matched with spice cli version by @Sevenannn in https://github.com/spiceai/spiceai/pull/4761
• Fix go lint errors by @sgrebnov in https://github.com/spiceai/spiceai/pull/5147
• Make DuckDB acceleration E2E tests more comprehensive by @sgrebnov in https://github.com/spiceai/spiceai/pull/5146
• Enable Spice to load larger than memory datasets into DuckDB accelerations by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5149
• Add temp_directory runtime parameter and insert it for DuckDB accelerations by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5152
• Fix Postgres and MySQL installation on macos14-runner (E2E CI) by @sgrebnov in https://github.com/spiceai/spiceai/pull/5155
• Enable E2E for DuckDB full mode acceleration with indexes only in CI by @sgrebnov in https://github.com/spiceai/spiceai/pull/5154

Dependencies​

No major dependency changes.

Changelog​

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.0.6...v1.0.7

The v0.12.1-alpha release introduces a new Snowflake data connector, support for UUID and TimestampTZ types in the PostgreSQL connector, and improved error messages across all data connectors. The Clickhouse data connector enables data streaming by default. The public SQL interface now restricts DML and DDL queries. Additionally, accelerated tables now fully support NULL values, and issues with schema conversion in these tables have been resolved.

Highlights​

• Snowflake Data Connector: Initial support for Snowflake as a data source.

• Clickhouse Data Streaming: Enables data streaming by default, eliminating in-memory result collection.

• Read-only SQL Interface: Disables DML (INSERT/UPDATE/DELETE) and DDL (CREATE/ALTER TABLE) queries for improved data source security.

• Error Message Improvements: Improved the error messages for commonly encountered issues with data connectors.

• Accelerated Tables: Supports NULL values across all data types and fixes schema conversion errors for consistent type handling.

Contributors​

• @ahirner
• @y-f-u
• @sgrebnov
• @ewgenius
• @Jeadie
• @gloomweaver
• @Sevenannn
• @digadeesh
• @phillipleblanc

What's Changed​

• Add schema types check for query result by @sgrebnov in https://github.com/spiceai/spiceai/pull/1212
• helm chart for v0.12.0-alpha by @y-f-u in https://github.com/spiceai/spiceai/pull/1235
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1232
• Bump spiceai version to v0.12.1-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1239
• Update ROADMAP.md - remove v0.12.0-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1241
• Raise errors in InsertBuilder by @Jeadie in https://github.com/spiceai/spiceai/pull/1242
• Update endgame template by @ewgenius in https://github.com/spiceai/spiceai/pull/1240
• Add E2E tests for acceleration engines types support by @sgrebnov in https://github.com/spiceai/spiceai/pull/1218
• Stream blocks to arrow by @gloomweaver in https://github.com/spiceai/spiceai/pull/1203
• Update enhancement.md to include a checklist item have a release notes entry for each enhancement. by @digadeesh in https://github.com/spiceai/spiceai/pull/1245
• arrow_sql_gen data column conversion by @Sevenannn in https://github.com/spiceai/spiceai/pull/1230
• Implement the Localhost Data Connector & fix DoPut by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1266
• Update postgres parameter check by @Sevenannn in https://github.com/spiceai/spiceai/pull/1244
• Record batch casting to fix SQLite data type issues by @y-f-u in https://github.com/spiceai/spiceai/pull/1261
• typo fix on Decimal in postgres arrow_sql_gen by @y-f-u in https://github.com/spiceai/spiceai/pull/1277
• Move verify_schema to arrow_tools by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1284
• Support UUID and TimestampTZ type for Postgres as Data Connector by @ahirner & @y-f-u https://github.com/spiceai/spiceai/pull/1276
• Fix linter warnings by @ewgenius in https://github.com/spiceai/spiceai/pull/1286
• Add Snowflake data connector by @sgrebnov in https://github.com/spiceai/spiceai/pull/1278
• Add Snowflake login support (username and password) by @sgrebnov in https://github.com/spiceai/spiceai/pull/1272
• convert timestamp properly in sql gen by @y-f-u in https://github.com/spiceai/spiceai/pull/1291
• Add if not exists clause to create statement on when creating a table using duckdb acceleration. by @digadeesh in https://github.com/spiceai/spiceai/pull/1290
• Disable DML & DDL queries in the public SQL interface by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1294
• Refactor duckdb to properly set access_mode for connection by @ewgenius in https://github.com/spiceai/spiceai/pull/1285
• do not insert batch for sqlite and postgres if no records in the record batch by @y-f-u in https://github.com/spiceai/spiceai/pull/1293
• Postgres - add custom error message for invalid error table by @ewgenius in https://github.com/spiceai/spiceai/pull/1295
• SQLite/Accelerators handle null values by @gloomweaver in https://github.com/spiceai/spiceai/pull/1298
• Add command to attach to running process by @gloomweaver in https://github.com/spiceai/spiceai/pull/1297
• Use the GITHUB_TOKEN environment variable in the installation script, if available, to avoid rate limiting in CI workflows by @ewgenius in https://github.com/spiceai/spiceai/pull/1302
• Fix unsupported SSL mode options for PostgreSQL connection string by @ewgenius in https://github.com/spiceai/spiceai/pull/1300
• Add CLI cmd spice login spark by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1303
• Check only the latest published release to avoid installing pre-release versions by @ewgenius in https://github.com/spiceai/spiceai/pull/1301
• Postgres data connector - handle invalid host/port and username/password errors by @ewgenius in https://github.com/spiceai/spiceai/pull/1292
• Fix the panic on bad clickhouse connection by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1306
• Improve Snowflake Data Connector by @sgrebnov https://github.com/spiceai/spiceai/pull/1296

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.12.0-alpha...v0.12.1-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

• Twitter: @spice_ai
• Discord: https://discord.gg/kZnTfneP5u
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai