Object Storage (S3)

Link AWS S3 (or S3-compatible) buckets to browse, upload and download files — and, on SQL Server 2022+ tenants, attach parquet/CSV files as queryable external tables. Lives under Settings → Object Storage (S3).

Connections: add a connection with a name, region, bucket, optional key prefix, optional custom endpoint (for S3-compatible storage — must be https:// and pass the same SSRF checks as a webhook), an access key id, and a secret access key. The secret is write-only — it is encrypted at rest and never shown again; leave it blank when editing to keep the stored one. Test probes the bucket. Managing connections needs data.ingest.manage; anyone with an ingest bucket can browse read-only.

Bucket browser: navigate folders with a breadcrumb, see each object’s size and modified time, upload files (streamed straight to S3, no memory buffering), download, and delete (with confirm). Large listings page with Load more. On attach-capable tenants, parquet/CSV rows also show an Attach as table action (link icon) — the fastest way to attach (see below).

Attach (SQL Server 2022+ only) — with schema auto-discovery: click Attach as table on a .parquet/.csv file in the browser and the attach dialog opens with the key and format prefilled and the columns detected automatically — parquet files are self-describing (the server reads just the file’s footer, never the whole object) and CSV headers are parsed with types inferred from a sample. You land in an editable column grid: untick columns to exclude them, rename or retype any column, and columns discovery couldn’t map (nested parquet lists/structs) are shown greyed-out with the reason. The table name defaults to the file name. For CSV files a Delimiter select appears, preselected to the delimiter detected in the file (comma, semicolon, tab or pipe) and overridable — so non-comma CSVs attach and read correctly. For a hand-entered object key or prefix, the Detect schema button does the same; you can still type every column manually if you prefer. The connector creates a PolyBase external table in a per-connection schema (s3_<id>, named from the connection’s immutable numeric id — renaming the connection later never renames or orphans the attached tables) that you can then query like any table. The attached-tables list shows what’s linked, with Detach (drops the external table; the S3 data is untouched). Attaching a table name that is already attached is refused with a clear message — detach it first or pick another name. Rotating a connection’s AWS keys (saving a new secret) also refreshes the SQL-side credential of any attached tables, so attached tables keep working after a key rotation. On SQL 2017/2019 tenants the attach panel is replaced by an inline note — “Attaching S3 files as external tables requires SQL Server 2022 or later.” — browse/upload/download still work everywhere.

Bucket locked down by IP? PolyBase reads S3 from the SQL Server’s public IP, not the Data Lake app’s — an aws:SourceIp-restricted bucket policy must also allow the SQL server’s egress IP. The tell-tale symptom: the bucket browser works fine, but attaching (or querying an attached table) fails with “content of directory cannot be listed”.

Exporting a table into a bucket: dlake s3 export <connection> <table> --format csv|parquet (or the admin-plane export_to_s3 tool) runs the normal scope-enforced export pipeline server-side and drops the finished file straight into the bucket.

Scoping keys to external tables: attached external tables appear in the API-key scope grid (Settings → Data Engine (DAB) → API Keys) under a “Lake / external tables” group, shown schema-qualified (e.g. s3_sales.Orders) with an S3-backed hint. They are read-only surfaces — only the Read action (and its column-level field policy) applies. A scoped key reads them through the raw-SQL tools (with the AllowRawSql opt-in), where access is database-enforced by the key’s SQL principal; the external tables are never exposed through the Data Engine (DAB) API itself. A lake row never grants a working-schema table of the same name (and vice versa).