Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/add union data #44

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

Feature/add union data #44

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
f53d024
started
fivetran-jamie Dec 15, 2023
e4f50e0
test and docs
fivetran-jamie Dec 19, 2023
3c2b394
Docs and sources
fivetran-jamie Dec 19, 2023
457d977
missed window ffunction
fivetran-jamie Dec 26, 2023
78fd1b4
readme docs for unioning sources
fivetran-jamie Dec 26, 2023
cf909d9
docs
fivetran-jamie Dec 26, 2023
25137a4
changelog
fivetran-jamie Dec 26, 2023
abb7bf2
update package versions
fivetran-jamie Dec 27, 2023
55e4b44
joe feedback
fivetran-jamie Dec 28, 2023
3639626
readme tweaks following hubspot
fivetran-jamie Jan 8, 2024
bd7f19a
Merge branch 'main' into feature/add-union-data
fivetran-jamie Jan 30, 2024
e76b12b
update readme
fivetran-jamie Jan 31, 2024
80dbfa4
swap source definition template
fivetran-jamie Feb 5, 2024
e53d94d
merge
fivetran-jamie Nov 6, 2024
8be3f39
working
fivetran-jamie Nov 7, 2024
a14fc9f
fix source relation
fivetran-jamie Nov 8, 2024
af6cad5
fix
fivetran-jamie Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# dbt_zendesk_source v0.11.0

## 🎉 Feature Update 🎉
This release supports running the package on multiple Zendesk sources at once! See the [README](https://github.com/fivetran/dbt_zendesk_source?tab=readme-ov-file#step-3-define-database-and-schema-variables) for details on how to leverage this feature ([PR #44](https://github.com/fivetran/dbt_zendesk_source/pull/44)).

# dbt_zendesk_source v0.10.0
[PR #42](https://github.com/fivetran/dbt_zendesk_source/pull/42) introduces the following updates:

Expand Down
373 changes: 371 additions & 2 deletions README.md
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion dbt_project.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
config-version: 2
name: 'zendesk_source'
version: '0.10.0'
version: '0.11.0'
require-dbt-version: [">=1.3.0", "<2.0.0"]
models:
zendesk_source:
Expand Down
6 changes: 4 additions & 2 deletions models/src_zendesk.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,18 @@ version: 2

sources:
- name: zendesk
schema: "{{var ('zendesk_schema', 'zendesk')}}"
schema: "{{ var('zendesk_schema', 'zendesk') }}"
database: "{% if target.type != 'spark'%}{{ var('zendesk_database', target.database) }}{% endif %}"
loader: fivetran
loaded_at_field: _fivetran_synced

config:
enabled: "{{ var('zendesk_union_schemas', []) == [] and var('zendesk_union_databases', []) == [] }}"

freshness:
warn_after: {count: 72, period: hour}
error_after: {count: 168, period: hour}


tables:
- name: ticket
identifier: "{{ var('zendesk_ticket_identifier', 'ticket')}}"
Expand Down
132 changes: 117 additions & 15 deletions models/stg_zendesk.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,15 @@ models:
description: >
Tickets are the means through which your end users (customers) communicate with agents in Zendesk Support. Tickets can
originate from a number of channels, including email, Help Center, chat, phone call, Twitter, Facebook, or the API.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- ticket_id
- source_relation
columns:
- name: ticket_id
description: Automatically assigned when the ticket is created
tests:
- unique
- not_null
- name: url
description: The API url of this ticket
Expand Down Expand Up @@ -58,17 +62,25 @@ models:
- name: source_to_address
description: The address of the source the ticket was created from
- name: source_to_name
description: The name of the source the ticket was created from
description: The name of the source the ticket was created from
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__brand
description: >
Brands are your customer-facing identities. They might represent multiple products or services, or they
might literally be multiple brands owned and represented by your company.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- brand_id
- source_relation
columns:
- name: brand_id
description: The ID automatically assigned when the brand is created
tests:
- unique
- not_null
- name: brand_url
description: The url of the brand
Expand All @@ -78,6 +90,10 @@ models:
description: The subdomain of the brand
- name: active
description: If the brand is set as active
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__domain_name
description: Domain names associated with an organization. An organization may have multiple domain names.
Expand All @@ -90,21 +106,33 @@ models:
description: The name of the domain associated with the organization
- name: index
description: Index number of the domain name associated with the organization
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__group
description: >
When support requests arrive in Zendesk Support, they can be assigned to a Group. Groups serve as the core
element of ticket workflow; support agents are organized into Groups and tickets can be assigned to a Group
only, or to an assigned agent within a Group. A ticket can never be assigned to an agent without also being
assigned to a Group.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- group_id
- source_relation
columns:
- name: group_id
description: Automatically assigned when creating groups
tests:
- unique
- not_null
- name: name
description: The name of the group
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__organization_tag
description: The tags associated with an organization. An organization may have multiple tags.
Expand All @@ -113,31 +141,47 @@ models:
description: Reference to the organization
- name: tag
description: Tag associated with the organization
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__organization
description: >
Just as agents can be segmented into groups in Zendesk Support, your customers (end-users) can be segmented into
organizations. You can manually assign customers to an organization or automatically assign them to an organization
by their email address domain. Organizations can be used in business rules to route tickets to groups of agents or
to send email notifications.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- organization_id
- source_relation
columns:
- name: organization_id
description: Automatically assigned when the organization is created
tests:
- unique
- not_null
- name: name
description: A unique name for the organization
- name: details
description: Any details obout the organization, such as the address
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__ticket_comment
description: Ticket comments represent the conversation between requesters, collaborators, and agents. Comments can be public or private.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- ticket_comment_id
- source_relation
columns:
- name: ticket_comment_id
description: Automatically assigned when the comment is created
tests:
- unique
- not_null
- name: body
description: The comment string
Expand All @@ -155,6 +199,10 @@ models:
description: Boolean field indicating if the comment is a twitter tweet
- name: is_voice_comment
description: Boolean field indicating if the comment is a voice comment
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__user_tag
description: Table containing all tags associated with a user. Only present if your account has user tagging enabled.
Expand All @@ -163,14 +211,22 @@ models:
description: Reference to the user
- name: tag
description: Tag associated with the user
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__user
description: Zendesk has three types of users, end-users (your customers), agents, and administrators.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- user_id
- source_relation
columns:
- name: user_id
description: Automatically assigned when the user is created
tests:
- unique
- not_null
- name: email
description: The user's primary email address. *Writeable on create only. On update, a secondary email is added. See Email Address
Expand All @@ -188,12 +244,18 @@ models:
description: The user's time zone. See Time Zone
- name: ticket_restriction
description: Specifies which tickets the user has access to. Possible values are organization, groups, assigned, requested and null
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__schedule
description: The support schedules created with different business hours and holidays.
description: The support schedules created with different business hours and holidays.
columns:
- name: schedule_id
description: ID automatically assigned to the schedule upon creation
tests:
- not_null
- name: schedule_name
description: Name of the schedule
- name: created_at
Expand All @@ -204,7 +266,11 @@ models:
description: End time of the schedule, in the schedule's time zone.
- name: time_zone
description: Timezone in which the schedule operates.

- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__ticket_schedule
description: The schedules applied to tickets through a trigger.
columns:
Expand All @@ -214,7 +280,11 @@ models:
description: The time the schedule was assigned to the ticket
- name: schedule_id
description: The ID of the schedule applied to the ticket

- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__ticket_form_history
description: Ticket forms allow an admin to define a subset of ticket fields for display to both agents and end users.
columns:
Expand All @@ -232,6 +302,10 @@ models:
description: If the form is set as active
- name: name
description: The name of the form
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__ticket_tag
description: >
Expand All @@ -242,7 +316,10 @@ models:
description: The ID of the ticket associated with the tag
- name: tags
description: The tag, or word(s), associated with the ticket

- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__ticket_field_history
description: All fields and field values associated with tickets.
Expand All @@ -259,6 +336,10 @@ models:
description: The value of the field
- name: user_id
description: The id of the user who made the update
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__daylight_time
description: >
Expand All @@ -267,7 +348,8 @@ models:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- time_zone
- year
- year
- source_relation
columns:
- name: daylight_end_utc
description: UTC timestamp of when Daylight Time ended in this year.
Expand All @@ -281,33 +363,53 @@ models:
description: Year in which daylight savings occurred.
- name: daylight_offset_minutes
description: Number of **minutes** added during Daylight Savings Time.
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__time_zone
description: Offsets (from UTC) for each timezone.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- time_zone
- source_relation
Comment on lines +412 to +416
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is resulting in a failure when running on the zendesk_yashwanth schema. Do you have any understanding why this is failing?

image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is the result of the time_zone name creating a struct when we don't want it to. I recall you had a fix for this. Do you know if that may have been missed for some reason?

image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be that we need to add the alias to the union_data macro here.

So it would be:

{%- if table_exists -%}
    select
        {{ dbt_utils.star(from=relation.value) }}
    from {{ relation.value }} as {{ table_identifier }}_table

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yes i missed adding this to the single-connector use-case inside union_data

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that did the trick!

columns:
- name: time_zone
description: Name of the time zone.
tests:
- unique
- not_null
- name: standard_offset
description: Standard offset of the timezone (non-daylight savings hours). In `+/-hh:mm` format.
- name: standard_offset_minutes
description: Standard offset of the timezone (non-daylight savings hours) in minutes.
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.

- name: stg_zendesk__schedule_holiday
description: Information about holidays for each specified schedule.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- holiday_id
- source_relation
columns:
- name: end_date_at
description: ISO 8601 representation of the holiday end date.
- name: holiday_id
description: The ID of the scheduled holiday.
tests:
- unique
- not_null
- name: holiday_name
description: Name of the holiday.
- name: schedule_id
description: The ID of the schedule.
- name: start_date_at
description: ISO 8601 representation of the holiday start date.
description: ISO 8601 representation of the holiday start date.
- name: source_relation
description: >
The schema or database this record came from if you are unioning multiple Zendesk connectors together in this package.
Empty string if you are not.
9 changes: 8 additions & 1 deletion models/stg_zendesk__brand.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ fields as (
staging_columns=get_brand_columns()
)
}}

{{ fivetran_utils.source_relation(
union_schema_variable='zendesk_union_schemas',
union_database_variable='zendesk_union_databases')
}}

from base
),
Expand All @@ -32,7 +37,9 @@ final as (
brand_url,
name,
subdomain,
active as is_active
active as is_active,
source_relation

from fields
where not coalesce(_fivetran_deleted, false)
)
Expand Down
Loading