Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Undefined date csv #316

Closed
wants to merge 3 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 24 additions & 14 deletions server/routes/utils/dataReportHelpers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,29 +7,29 @@
import converter from 'json-2-csv';
import _ from 'lodash';
import moment from 'moment-timezone';
import { DATA_REPORT_CONFIG } from './constants';

Check failure on line 10 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

'DATA_REPORT_CONFIG' is defined but never used. Allowed unused vars must match /^_/u
import {
buildOpenSearchQuery,
Filter,
Query,
OpenSearchQueryConfig,
} from '../../../../../src/plugins/data/common';
import { string } from 'joi';

Check failure on line 17 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

`joi` import should occur before import of `./constants`

Check failure on line 17 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

'string' is defined but never used. Allowed unused vars must match /^_/u

export var metaData = {

Check failure on line 19 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected var, use let or const instead
saved_search_id: <string>null,

Check failure on line 20 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Use 'as string' instead of '<string>'
report_format: <string>null,

Check failure on line 21 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Use 'as string' instead of '<string>'
start: <string>null,

Check failure on line 22 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Use 'as string' instead of '<string>'
end: <string>null,

Check failure on line 23 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Use 'as string' instead of '<string>'
fields: <string>null,

Check failure on line 24 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Use 'as string' instead of '<string>'
type: <string>null,

Check failure on line 25 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Use 'as string' instead of '<string>'
timeFieldName: <string>null,
sorting: <string>null,
fields_exist: <boolean>false,
selectedFields: <any>[],

Check warning on line 29 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected any. Specify a different type
paternName: <string>null,
searchSourceJSON: <any>[],

Check warning on line 31 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected any. Specify a different type
dateFields: <any>[],

Check warning on line 32 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected any. Specify a different type
};

// Get the selected columns by the user.
Expand All @@ -52,7 +52,7 @@
// Build the OpenSearch query from the meta data
// is_count is set to 1 if we building the count query but 0 if we building the fetch data query
export const buildRequestBody = (
report: any,

Check warning on line 55 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected any. Specify a different type
allowLeadingWildcards: boolean,
is_count: number
) => {
Expand Down Expand Up @@ -119,13 +119,13 @@

// Fetch the data from OpenSearch
export const getOpenSearchData = (
arrayHits: any,

Check warning on line 122 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected any. Specify a different type
report: { _source: any },

Check warning on line 123 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected any. Specify a different type
params: { excel: any; limit: number },

Check warning on line 124 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected any. Specify a different type
dateFormat: string,
timezone: string
) => {
let hits: any = [];

Check warning on line 128 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected any. Specify a different type
for (let valueRes of arrayHits) {
for (let data of valueRes.hits) {
const fields = data.fields;
Expand All @@ -135,21 +135,27 @@
let keys;
keys = dateField.split('.');
const dateValue = data._source[dateField];
const fieldDateValue = fields[dateField];
const isDateFieldPresent = isKeyPresent(data._source, dateField);

if (isDateFieldPresent) {
const fieldDateValue = fields[dateField];
// if its not a nested date field
if (keys.length === 1) {
// if conditions to determine if the date field's value is an array or a string
if (typeof dateValue === 'string') {
data._source[keys] = moment.utc(dateValue).tz(timezone).format(dateFormat);
} else if (
dateValue.length !== 0 &&
dateValue instanceof Array
if (
typeof dateValue === 'string' ||
typeof dateValue === 'number'
) {
data._source[keys] = moment
.utc(dateValue)
.tz(timezone)
.format(dateFormat);
} else if (dateValue.length !== 0 && dateValue instanceof Array) {
fieldDateValue.forEach((element, index) => {
data._source[keys][index] = moment.utc(element).tz(timezone).format(dateFormat);
data._source[keys][index] = moment
.utc(element)
.tz(timezone)
.format(dateFormat);
});
} else {
data._source[keys] = [];
Expand All @@ -158,15 +164,19 @@
} else {
let keyElement = keys.shift();
// if conditions to determine if the date field's value is an array or a string
if (typeof fieldDateValue === 'string') {
keys.push(moment.utc(fieldDateValue).tz(timezone).format(dateFormat));
} else if (
dateValue.length !== 0 &&
dateValue instanceof Array
if (
typeof fieldDateValue === 'string' ||
typeof fieldDateValue === 'number'
) {
keys.push(
moment.utc(fieldDateValue).tz(timezone).format(dateFormat)
);
} else if (dateValue.length !== 0 && dateValue instanceof Array) {
let tempArray: string[] = [];
fieldDateValue.forEach((index) => {
tempArray.push(moment.utc(index).tz(timezone).format(dateFormat));
tempArray.push(
moment.utc(index).tz(timezone).format(dateFormat)
);
});
keys.push(tempArray);
} else {
Expand Down Expand Up @@ -205,7 +215,7 @@

//Convert the data to Csv format
export const convertToCSV = async (dataset, csvSeparator) => {
let convertedData: any = [];

Check warning on line 218 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected any. Specify a different type
const options = {
delimiter: { field: csvSeparator, eol: '\n' },
emptyFieldValue: ' ',
Expand Down Expand Up @@ -254,7 +264,7 @@
* This is intend to avoid CSV injection in Microsoft Excel.
* @param doc document
*/
function sanitize(doc: any) {

Check warning on line 267 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint

Unexpected any. Specify a different type
for (const field in doc) {
if (doc[field] == null) continue;
if (
Expand Down Expand Up @@ -311,4 +321,4 @@
docvalue_fields: docValues,
};
return requestBody;
};
};
Loading