How to add complex calculations to Elastic SearchHow often does the Impala StateStore refresh?How does...
Saint abbreviation
Building an exterior wall within an exterior wall for insulation
What game did these black and yellow dice come from?
What happened to my GE option?
How to assess the long-term stability of a college as part of a job search
Plausible reason for gold-digging ant
What happens when I Twin Life Transference?
Separate environment for personal and development use under macOS
Explanation of a regular pattern only occuring for prime numbers
Why is Agricola named as such?
What will happen if I transfer large sums of money into my bank account from a pre-paid debit card or gift card?
Can I announce prefix 161.117.25.0/24 even though I don't have all of /24 IPs
How much mayhem could I cause as a fish?
What is the wife of a henpecked husband called?
Why did Luke use his left hand to shoot?
Why did the villain in the first Men in Black movie care about Earth's Cockroaches?
Can 5 Aarakocra PCs summon an Air Elemental?
Why didn't Tom Riddle take the presence of Fawkes and the Sorting Hat as more of a threat?
Why do neural networks need so many training examples to perform?
Is there a lava-breathing lizard creature (that could be worshipped by a cult) in 5e?
What is a good reason for every spaceship to carry a weapon on board?
Why does magnet wire need to be insulated?
TikZ graph edges not drawn nicely
How would an AI self awareness kill switch work?
How to add complex calculations to Elastic Search
How often does the Impala StateStore refresh?How does PolyBase collect statistics from external tables?Synchronize between two elastic-search for redundancyHow we can do full text search and facets stuffs on Bigquery?JDBC Sqlserver to elastic search : one input to different output?How to installation spark standalone mode in ubuntuHow to insert several csv files into Elasticsearch?Combine relational DB and Elastic searchElastic: Are Analyzers only for `text` type fields?
I'm using elastic search to index large amounts of sensor data for analytics purposes. The table has 4 million + rows and growing fast - expecting 40 million within the next year. This makes Elastic Search seem like a natural fit, especially with tools such as Kibana to easily display the data.
Elastic Search seems great, however there are are some more complex calculations that have to be performed as well. One such calculation is for our "average user time", where we take two data points (timestamp of item picked up and timestamp of item placed back), subtract them from each other and do an average of all these for one specific customer over a specific timeframe. The SQL query would look something like "select * from events where event_type = 'object picked up' or event_type = 'object placed back down'" then take all these events and get diffs on all their timestamps, add them all together then divide by count.
These types of calculations to my understanding are not the type of thing that Elastic Search is meant to do. I've had people recommend Hadoop but that could take a long time to get set up (we're a startup and have clients that need this info asap) and we can use a fast language like GO or Node/ Javascript to batch process things and add them to the DB periodically... but what is the RIGHT way to do this? Allowing for future scalability and working nicely with Elastic Search.
Our setup is: Rails, AngularJS, Elastic Search, Heroku, Postgres.
hadoop elasticsearch
asked Oct 30 '15 at 2:58
RobRob
1011
bumped to the homepage by Community♦ 49 secs ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
I'm using elastic search to index large amounts of sensor data for analytics purposes. The table has 4 million + rows and growing fast - expecting 40 million within the next year. This makes Elastic Search seem like a natural fit, especially with tools such as Kibana to easily display the data.
Elastic Search seems great, however there are are some more complex calculations that have to be performed as well. One such calculation is for our "average user time", where we take two data points (timestamp of item picked up and timestamp of item placed back), subtract them from each other and do an average of all these for one specific customer over a specific timeframe. The SQL query would look something like "select * from events where event_type = 'object picked up' or event_type = 'object placed back down'" then take all these events and get diffs on all their timestamps, add them all together then divide by count.
These types of calculations to my understanding are not the type of thing that Elastic Search is meant to do. I've had people recommend Hadoop but that could take a long time to get set up (we're a startup and have clients that need this info asap) and we can use a fast language like GO or Node/ Javascript to batch process things and add them to the DB periodically... but what is the RIGHT way to do this? Allowing for future scalability and working nicely with Elastic Search.
Our setup is: Rails, AngularJS, Elastic Search, Heroku, Postgres.
hadoop elasticsearch
asked Oct 30 '15 at 2:58
RobRob
1011
bumped to the homepage by Community♦ 49 secs ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
I'm using elastic search to index large amounts of sensor data for analytics purposes. The table has 4 million + rows and growing fast - expecting 40 million within the next year. This makes Elastic Search seem like a natural fit, especially with tools such as Kibana to easily display the data.
Elastic Search seems great, however there are are some more complex calculations that have to be performed as well. One such calculation is for our "average user time", where we take two data points (timestamp of item picked up and timestamp of item placed back), subtract them from each other and do an average of all these for one specific customer over a specific timeframe. The SQL query would look something like "select * from events where event_type = 'object picked up' or event_type = 'object placed back down'" then take all these events and get diffs on all their timestamps, add them all together then divide by count.
These types of calculations to my understanding are not the type of thing that Elastic Search is meant to do. I've had people recommend Hadoop but that could take a long time to get set up (we're a startup and have clients that need this info asap) and we can use a fast language like GO or Node/ Javascript to batch process things and add them to the DB periodically... but what is the RIGHT way to do this? Allowing for future scalability and working nicely with Elastic Search.
Our setup is: Rails, AngularJS, Elastic Search, Heroku, Postgres.
hadoop elasticsearch
asked Oct 30 '15 at 2:58
RobRob
1011
I'm using elastic search to index large amounts of sensor data for analytics purposes. The table has 4 million + rows and growing fast - expecting 40 million within the next year. This makes Elastic Search seem like a natural fit, especially with tools such as Kibana to easily display the data.
Elastic Search seems great, however there are are some more complex calculations that have to be performed as well. One such calculation is for our "average user time", where we take two data points (timestamp of item picked up and timestamp of item placed back), subtract them from each other and do an average of all these for one specific customer over a specific timeframe. The SQL query would look something like "select * from events where event_type = 'object picked up' or event_type = 'object placed back down'" then take all these events and get diffs on all their timestamps, add them all together then divide by count.
These types of calculations to my understanding are not the type of thing that Elastic Search is meant to do. I've had people recommend Hadoop but that could take a long time to get set up (we're a startup and have clients that need this info asap) and we can use a fast language like GO or Node/ Javascript to batch process things and add them to the DB periodically... but what is the RIGHT way to do this? Allowing for future scalability and working nicely with Elastic Search.
Our setup is: Rails, AngularJS, Elastic Search, Heroku, Postgres.
hadoop elasticsearch
hadoop elasticsearch
asked Oct 30 '15 at 2:58
RobRob
1011
asked Oct 30 '15 at 2:58
RobRob
1011
asked Oct 30 '15 at 2:58
RobRob
1011
asked Oct 30 '15 at 2:58
RobRob
1011
asked Oct 30 '15 at 2:58
RobRob
1011
1011
bumped to the homepage by Community♦ 49 secs ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 49 secs ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
My experience is that some calculations you can perform using "aggregations" feature from elasticseach:
- Elasticsearch Aggregations
- An Introduction to Elasticsearch Aggregations
But at some point there is only the way to get some counters - results from queries that gives you not really the records, but numbers describing them - and preform calculations in separate own application using Elasticsearch API.
answered Sep 26 '16 at 9:19
Seweryn Habdank-WojewódzkiSeweryn Habdank-Wojewódzki
1012
Please add the main points from those links. If they disappear, your answer will be much less useful.
– dezso
Sep 26 '16 at 10:06
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f119622%2fhow-to-add-complex-calculations-to-elastic-search%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
My experience is that some calculations you can perform using "aggregations" feature from elasticseach:
- Elasticsearch Aggregations
- An Introduction to Elasticsearch Aggregations
But at some point there is only the way to get some counters - results from queries that gives you not really the records, but numbers describing them - and preform calculations in separate own application using Elasticsearch API.
answered Sep 26 '16 at 9:19
Seweryn Habdank-WojewódzkiSeweryn Habdank-Wojewódzki
1012
Please add the main points from those links. If they disappear, your answer will be much less useful.
– dezso
Sep 26 '16 at 10:06
add a comment |
My experience is that some calculations you can perform using "aggregations" feature from elasticseach:
- Elasticsearch Aggregations
- An Introduction to Elasticsearch Aggregations
But at some point there is only the way to get some counters - results from queries that gives you not really the records, but numbers describing them - and preform calculations in separate own application using Elasticsearch API.
answered Sep 26 '16 at 9:19
Seweryn Habdank-WojewódzkiSeweryn Habdank-Wojewódzki
1012
Please add the main points from those links. If they disappear, your answer will be much less useful.
– dezso
Sep 26 '16 at 10:06
add a comment |
My experience is that some calculations you can perform using "aggregations" feature from elasticseach:
- Elasticsearch Aggregations
- An Introduction to Elasticsearch Aggregations
But at some point there is only the way to get some counters - results from queries that gives you not really the records, but numbers describing them - and preform calculations in separate own application using Elasticsearch API.
answered Sep 26 '16 at 9:19
Seweryn Habdank-WojewódzkiSeweryn Habdank-Wojewódzki
1012
My experience is that some calculations you can perform using "aggregations" feature from elasticseach:
- Elasticsearch Aggregations
- An Introduction to Elasticsearch Aggregations
But at some point there is only the way to get some counters - results from queries that gives you not really the records, but numbers describing them - and preform calculations in separate own application using Elasticsearch API.
answered Sep 26 '16 at 9:19
Seweryn Habdank-WojewódzkiSeweryn Habdank-Wojewódzki
1012
answered Sep 26 '16 at 9:19
Seweryn Habdank-WojewódzkiSeweryn Habdank-Wojewódzki
1012
answered Sep 26 '16 at 9:19
Seweryn Habdank-WojewódzkiSeweryn Habdank-Wojewódzki
1012
answered Sep 26 '16 at 9:19
Seweryn Habdank-WojewódzkiSeweryn Habdank-Wojewódzki
1012
1012
Please add the main points from those links. If they disappear, your answer will be much less useful.
– dezso
Sep 26 '16 at 10:06
add a comment |
Please add the main points from those links. If they disappear, your answer will be much less useful.
– dezso
Sep 26 '16 at 10:06
Please add the main points from those links. If they disappear, your answer will be much less useful.
– dezso
Sep 26 '16 at 10:06
Please add the main points from those links. If they disappear, your answer will be much less useful.
– dezso
Sep 26 '16 at 10:06
add a comment |
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f119622%2fhow-to-add-complex-calculations-to-elastic-search%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
