What are the disadvantages of having a left skewed distribution?2019 Community Moderator ElectionHow to deal...
How much RAM could one put in a typical 80386 setup?
Languages that we cannot (dis)prove to be Context-Free
Why doesn't a class having private constructor prevent inheriting from this class? How to control which classes can inherit from a certain base?
Alternative to sending password over mail?
Is it unprofessional to ask if a job posting on GlassDoor is real?
How can bays and straits be determined in a procedurally generated map?
Important Resources for Dark Age Civilizations?
How much of data wrangling is a data scientist's job?
Perform and show arithmetic with LuaLaTeX
Maximum likelihood parameters deviate from posterior distributions
Rock identification in KY
What does "Puller Prush Person" mean?
Replacing matching entries in one column of a file by another column from a different file
What typically incentivizes a professor to change jobs to a lower ranking university?
What's that red-plus icon near a text?
Is it inappropriate for a student to attend their mentor's dissertation defense?
What is a clear way to write a bar that has an extra beat?
Watching something be written to a file live with tail
Could an aircraft fly or hover using only jets of compressed air?
Client team has low performances and low technical skills: we always fix their work and now they stop collaborate with us. How to solve?
How is it possible to have an ability score that is less than 3?
How to format long polynomial?
Is it possible to do 50 km distance without any previous training?
Did Shadowfax go to Valinor?
What are the disadvantages of having a left skewed distribution?
2019 Community Moderator ElectionHow to deal with a skewed data-set having all the samples almost similar?What are the “extra nodes” in XGboost?What are the disadvantages of Azure's ML vs a pure code approach (R/SKlearn)What are the tools to plot cluster results?What is the best way to normalize histogram vectors to get distribution?What are the benefits of having ML in js?What data treatment/transformation should be applied if there are a lot of outliers and features lack normal distribution?What are the best practices for data formatting?What are the assumptions of linear regressionHistogram is extremely skewed to the left
$begingroup$
I'm currently working on a classification problem and I've a numerical column which is left skewed. i've read many posts where people are recommending to take log transformation or boxcox transformation to fix the left skewness.
So I was wondering what would happen If I left the skewness as it is and continue with my model building? Are there any advantages of fixing skewness for classification problem (knn, logistic regression)?
machine-learning python
asked 1 hour ago
user214user214
20417
$endgroup$
add a comment |
$begingroup$
I'm currently working on a classification problem and I've a numerical column which is left skewed. i've read many posts where people are recommending to take log transformation or boxcox transformation to fix the left skewness.
So I was wondering what would happen If I left the skewness as it is and continue with my model building? Are there any advantages of fixing skewness for classification problem (knn, logistic regression)?
machine-learning python
asked 1 hour ago
user214user214
20417
$endgroup$
add a comment |
$begingroup$
I'm currently working on a classification problem and I've a numerical column which is left skewed. i've read many posts where people are recommending to take log transformation or boxcox transformation to fix the left skewness.
So I was wondering what would happen If I left the skewness as it is and continue with my model building? Are there any advantages of fixing skewness for classification problem (knn, logistic regression)?
machine-learning python
asked 1 hour ago
user214user214
20417
$endgroup$
I'm currently working on a classification problem and I've a numerical column which is left skewed. i've read many posts where people are recommending to take log transformation or boxcox transformation to fix the left skewness.
So I was wondering what would happen If I left the skewness as it is and continue with my model building? Are there any advantages of fixing skewness for classification problem (knn, logistic regression)?
machine-learning python
machine-learning python
asked 1 hour ago
user214user214
20417
asked 1 hour ago
user214user214
20417
asked 1 hour ago
user214user214
20417
asked 1 hour ago
user214user214
20417
asked 1 hour ago
user214user214
20417
20417
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
There are issues that will depend on specific features of your data and analytic approach, but in general skewed data (in either direction) will degrade some of your model's ability to describe more "typical" cases in order to deal with much rarer cases which happen to take extreme values.
Since "typical" cases are more common than extreme ones in a skewed data set, you are losing some precision with the cases you'll see most often in order to accommodate cases that you'll see only rarely. Determining a coefficient for a thousand observations which are all between [0,10] is likely to be more precise than for 990 observations between [0,10] and 10 observations between [1,000, 1,000,000]. This can lead to your model being less useful overall.
"Fixing" skewness can provide a variety of benefits, including making analysis which depends on the data being approximately Normally distributed possible/more informative. It can also produce results which are reported on a sensible scale (this is very situation-dependent), and prevent extreme values (relative to other predictors) from over- or underestimating the influence of the skewed predictor on the predicted classification.
You can test this somewhat (in a non-definitive way, to be sure) by training models with varying subsets of your data: everything you've got, just as it is, your data without that skewed variable, your data with that variable but excluding values outside of the "typical" range (though you'll have to be careful in defining that), your data with the skewed variable distribution transformed or re-scaled, etc.
As for fixing it, transformations and re-scaling often make sense. But I cannot emphasize enough:
Fiddling with variables and their distributions should follow from properties of those variables, not your convenience in modelling.
Log-transforming skewed variables is a prime example of this:
- If you really think that a variable operates on a geometric scale,
and you want your model to operate on an arithmetic scale, then log
transformation can make a lot of sense. - If you think that variable operates on an arithmetic scale, but you
find its distribution inconvenient and think a log transformation
would produce a more convenient distribution, it may make sense to
transform. It will change how the model is used and interpreted,
usually making it more dense and harder to interpret clearly, but
that may or may not be worthwhile. For example, if you take the log of a numeric outcome and the log of a numeric predictor, the result has to be interpreted as an elasticity between them, which can be awkward to work with and is often not what is desired. - If you think that a log transformation would be desirable for a
variable, but it has a lot of observations with a value of 0, then
log transformation isn't really an option for you, whether it would
be convenient or not. (Adding a "small value" to the 0 observations
causes lots of problems-- take the logs of 1-10, and then 0.0 to
1.0).
answered 36 mins ago
Upper_CaseUpper_Case
1312
$endgroup$
$begingroup$
Assume I've numeric column such as price and it's heavily left skewed. I'm thinking of using few basic classification algorithms. What should be my approach? Should I go for log transformation or boxcox transformation?
$endgroup$
– user214
28 mins ago
$begingroup$
@user214 Left-skewed price information? That sounds interesting! (My research data is generally skewed hard to the right). There is always variation between study contexts, but I generally think of money as "geometric enough" that a log transformation is appropriate (or at least strongly defensible). Whether or not that's the ideal transformation is a very difficult question to answer, but log transformation is unlikely to be a problem for you here. You'll just need to remember that anything about that predictor will be reported on a log scale, and interpret accordingly.
$endgroup$
– Upper_Case
22 mins ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48711%2fwhat-are-the-disadvantages-of-having-a-left-skewed-distribution%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
There are issues that will depend on specific features of your data and analytic approach, but in general skewed data (in either direction) will degrade some of your model's ability to describe more "typical" cases in order to deal with much rarer cases which happen to take extreme values.
Since "typical" cases are more common than extreme ones in a skewed data set, you are losing some precision with the cases you'll see most often in order to accommodate cases that you'll see only rarely. Determining a coefficient for a thousand observations which are all between [0,10] is likely to be more precise than for 990 observations between [0,10] and 10 observations between [1,000, 1,000,000]. This can lead to your model being less useful overall.
"Fixing" skewness can provide a variety of benefits, including making analysis which depends on the data being approximately Normally distributed possible/more informative. It can also produce results which are reported on a sensible scale (this is very situation-dependent), and prevent extreme values (relative to other predictors) from over- or underestimating the influence of the skewed predictor on the predicted classification.
You can test this somewhat (in a non-definitive way, to be sure) by training models with varying subsets of your data: everything you've got, just as it is, your data without that skewed variable, your data with that variable but excluding values outside of the "typical" range (though you'll have to be careful in defining that), your data with the skewed variable distribution transformed or re-scaled, etc.
As for fixing it, transformations and re-scaling often make sense. But I cannot emphasize enough:
Fiddling with variables and their distributions should follow from properties of those variables, not your convenience in modelling.
Log-transforming skewed variables is a prime example of this:
- If you really think that a variable operates on a geometric scale,
and you want your model to operate on an arithmetic scale, then log
transformation can make a lot of sense. - If you think that variable operates on an arithmetic scale, but you
find its distribution inconvenient and think a log transformation
would produce a more convenient distribution, it may make sense to
transform. It will change how the model is used and interpreted,
usually making it more dense and harder to interpret clearly, but
that may or may not be worthwhile. For example, if you take the log of a numeric outcome and the log of a numeric predictor, the result has to be interpreted as an elasticity between them, which can be awkward to work with and is often not what is desired. - If you think that a log transformation would be desirable for a
variable, but it has a lot of observations with a value of 0, then
log transformation isn't really an option for you, whether it would
be convenient or not. (Adding a "small value" to the 0 observations
causes lots of problems-- take the logs of 1-10, and then 0.0 to
1.0).
answered 36 mins ago
Upper_CaseUpper_Case
1312
$endgroup$
$begingroup$
Assume I've numeric column such as price and it's heavily left skewed. I'm thinking of using few basic classification algorithms. What should be my approach? Should I go for log transformation or boxcox transformation?
$endgroup$
– user214
28 mins ago
$begingroup$
@user214 Left-skewed price information? That sounds interesting! (My research data is generally skewed hard to the right). There is always variation between study contexts, but I generally think of money as "geometric enough" that a log transformation is appropriate (or at least strongly defensible). Whether or not that's the ideal transformation is a very difficult question to answer, but log transformation is unlikely to be a problem for you here. You'll just need to remember that anything about that predictor will be reported on a log scale, and interpret accordingly.
$endgroup$
– Upper_Case
22 mins ago
add a comment |
$begingroup$
There are issues that will depend on specific features of your data and analytic approach, but in general skewed data (in either direction) will degrade some of your model's ability to describe more "typical" cases in order to deal with much rarer cases which happen to take extreme values.
Since "typical" cases are more common than extreme ones in a skewed data set, you are losing some precision with the cases you'll see most often in order to accommodate cases that you'll see only rarely. Determining a coefficient for a thousand observations which are all between [0,10] is likely to be more precise than for 990 observations between [0,10] and 10 observations between [1,000, 1,000,000]. This can lead to your model being less useful overall.
"Fixing" skewness can provide a variety of benefits, including making analysis which depends on the data being approximately Normally distributed possible/more informative. It can also produce results which are reported on a sensible scale (this is very situation-dependent), and prevent extreme values (relative to other predictors) from over- or underestimating the influence of the skewed predictor on the predicted classification.
You can test this somewhat (in a non-definitive way, to be sure) by training models with varying subsets of your data: everything you've got, just as it is, your data without that skewed variable, your data with that variable but excluding values outside of the "typical" range (though you'll have to be careful in defining that), your data with the skewed variable distribution transformed or re-scaled, etc.
As for fixing it, transformations and re-scaling often make sense. But I cannot emphasize enough:
Fiddling with variables and their distributions should follow from properties of those variables, not your convenience in modelling.
Log-transforming skewed variables is a prime example of this:
- If you really think that a variable operates on a geometric scale,
and you want your model to operate on an arithmetic scale, then log
transformation can make a lot of sense. - If you think that variable operates on an arithmetic scale, but you
find its distribution inconvenient and think a log transformation
would produce a more convenient distribution, it may make sense to
transform. It will change how the model is used and interpreted,
usually making it more dense and harder to interpret clearly, but
that may or may not be worthwhile. For example, if you take the log of a numeric outcome and the log of a numeric predictor, the result has to be interpreted as an elasticity between them, which can be awkward to work with and is often not what is desired. - If you think that a log transformation would be desirable for a
variable, but it has a lot of observations with a value of 0, then
log transformation isn't really an option for you, whether it would
be convenient or not. (Adding a "small value" to the 0 observations
causes lots of problems-- take the logs of 1-10, and then 0.0 to
1.0).
answered 36 mins ago
Upper_CaseUpper_Case
1312
$endgroup$
$begingroup$
Assume I've numeric column such as price and it's heavily left skewed. I'm thinking of using few basic classification algorithms. What should be my approach? Should I go for log transformation or boxcox transformation?
$endgroup$
– user214
28 mins ago
$begingroup$
@user214 Left-skewed price information? That sounds interesting! (My research data is generally skewed hard to the right). There is always variation between study contexts, but I generally think of money as "geometric enough" that a log transformation is appropriate (or at least strongly defensible). Whether or not that's the ideal transformation is a very difficult question to answer, but log transformation is unlikely to be a problem for you here. You'll just need to remember that anything about that predictor will be reported on a log scale, and interpret accordingly.
$endgroup$
– Upper_Case
22 mins ago
add a comment |
$begingroup$
There are issues that will depend on specific features of your data and analytic approach, but in general skewed data (in either direction) will degrade some of your model's ability to describe more "typical" cases in order to deal with much rarer cases which happen to take extreme values.
Since "typical" cases are more common than extreme ones in a skewed data set, you are losing some precision with the cases you'll see most often in order to accommodate cases that you'll see only rarely. Determining a coefficient for a thousand observations which are all between [0,10] is likely to be more precise than for 990 observations between [0,10] and 10 observations between [1,000, 1,000,000]. This can lead to your model being less useful overall.
"Fixing" skewness can provide a variety of benefits, including making analysis which depends on the data being approximately Normally distributed possible/more informative. It can also produce results which are reported on a sensible scale (this is very situation-dependent), and prevent extreme values (relative to other predictors) from over- or underestimating the influence of the skewed predictor on the predicted classification.
You can test this somewhat (in a non-definitive way, to be sure) by training models with varying subsets of your data: everything you've got, just as it is, your data without that skewed variable, your data with that variable but excluding values outside of the "typical" range (though you'll have to be careful in defining that), your data with the skewed variable distribution transformed or re-scaled, etc.
As for fixing it, transformations and re-scaling often make sense. But I cannot emphasize enough:
Fiddling with variables and their distributions should follow from properties of those variables, not your convenience in modelling.
Log-transforming skewed variables is a prime example of this:
- If you really think that a variable operates on a geometric scale,
and you want your model to operate on an arithmetic scale, then log
transformation can make a lot of sense. - If you think that variable operates on an arithmetic scale, but you
find its distribution inconvenient and think a log transformation
would produce a more convenient distribution, it may make sense to
transform. It will change how the model is used and interpreted,
usually making it more dense and harder to interpret clearly, but
that may or may not be worthwhile. For example, if you take the log of a numeric outcome and the log of a numeric predictor, the result has to be interpreted as an elasticity between them, which can be awkward to work with and is often not what is desired. - If you think that a log transformation would be desirable for a
variable, but it has a lot of observations with a value of 0, then
log transformation isn't really an option for you, whether it would
be convenient or not. (Adding a "small value" to the 0 observations
causes lots of problems-- take the logs of 1-10, and then 0.0 to
1.0).
answered 36 mins ago
Upper_CaseUpper_Case
1312
$endgroup$
There are issues that will depend on specific features of your data and analytic approach, but in general skewed data (in either direction) will degrade some of your model's ability to describe more "typical" cases in order to deal with much rarer cases which happen to take extreme values.
Since "typical" cases are more common than extreme ones in a skewed data set, you are losing some precision with the cases you'll see most often in order to accommodate cases that you'll see only rarely. Determining a coefficient for a thousand observations which are all between [0,10] is likely to be more precise than for 990 observations between [0,10] and 10 observations between [1,000, 1,000,000]. This can lead to your model being less useful overall.
"Fixing" skewness can provide a variety of benefits, including making analysis which depends on the data being approximately Normally distributed possible/more informative. It can also produce results which are reported on a sensible scale (this is very situation-dependent), and prevent extreme values (relative to other predictors) from over- or underestimating the influence of the skewed predictor on the predicted classification.
You can test this somewhat (in a non-definitive way, to be sure) by training models with varying subsets of your data: everything you've got, just as it is, your data without that skewed variable, your data with that variable but excluding values outside of the "typical" range (though you'll have to be careful in defining that), your data with the skewed variable distribution transformed or re-scaled, etc.
As for fixing it, transformations and re-scaling often make sense. But I cannot emphasize enough:
Fiddling with variables and their distributions should follow from properties of those variables, not your convenience in modelling.
Log-transforming skewed variables is a prime example of this:
- If you really think that a variable operates on a geometric scale,
and you want your model to operate on an arithmetic scale, then log
transformation can make a lot of sense. - If you think that variable operates on an arithmetic scale, but you
find its distribution inconvenient and think a log transformation
would produce a more convenient distribution, it may make sense to
transform. It will change how the model is used and interpreted,
usually making it more dense and harder to interpret clearly, but
that may or may not be worthwhile. For example, if you take the log of a numeric outcome and the log of a numeric predictor, the result has to be interpreted as an elasticity between them, which can be awkward to work with and is often not what is desired. - If you think that a log transformation would be desirable for a
variable, but it has a lot of observations with a value of 0, then
log transformation isn't really an option for you, whether it would
be convenient or not. (Adding a "small value" to the 0 observations
causes lots of problems-- take the logs of 1-10, and then 0.0 to
1.0).
answered 36 mins ago
Upper_CaseUpper_Case
1312
answered 36 mins ago
Upper_CaseUpper_Case
1312
answered 36 mins ago
Upper_CaseUpper_Case
1312
answered 36 mins ago
Upper_CaseUpper_Case
1312
1312
$begingroup$
Assume I've numeric column such as price and it's heavily left skewed. I'm thinking of using few basic classification algorithms. What should be my approach? Should I go for log transformation or boxcox transformation?
$endgroup$
– user214
28 mins ago
$begingroup$
@user214 Left-skewed price information? That sounds interesting! (My research data is generally skewed hard to the right). There is always variation between study contexts, but I generally think of money as "geometric enough" that a log transformation is appropriate (or at least strongly defensible). Whether or not that's the ideal transformation is a very difficult question to answer, but log transformation is unlikely to be a problem for you here. You'll just need to remember that anything about that predictor will be reported on a log scale, and interpret accordingly.
$endgroup$
– Upper_Case
22 mins ago
add a comment |
$begingroup$
Assume I've numeric column such as price and it's heavily left skewed. I'm thinking of using few basic classification algorithms. What should be my approach? Should I go for log transformation or boxcox transformation?
$endgroup$
– user214
28 mins ago
$begingroup$
@user214 Left-skewed price information? That sounds interesting! (My research data is generally skewed hard to the right). There is always variation between study contexts, but I generally think of money as "geometric enough" that a log transformation is appropriate (or at least strongly defensible). Whether or not that's the ideal transformation is a very difficult question to answer, but log transformation is unlikely to be a problem for you here. You'll just need to remember that anything about that predictor will be reported on a log scale, and interpret accordingly.
$endgroup$
– Upper_Case
22 mins ago
$begingroup$
Assume I've numeric column such as price and it's heavily left skewed. I'm thinking of using few basic classification algorithms. What should be my approach? Should I go for log transformation or boxcox transformation?
$endgroup$
– user214
28 mins ago
$begingroup$
Assume I've numeric column such as price and it's heavily left skewed. I'm thinking of using few basic classification algorithms. What should be my approach? Should I go for log transformation or boxcox transformation?
$endgroup$
– user214
28 mins ago
$begingroup$
@user214 Left-skewed price information? That sounds interesting! (My research data is generally skewed hard to the right). There is always variation between study contexts, but I generally think of money as "geometric enough" that a log transformation is appropriate (or at least strongly defensible). Whether or not that's the ideal transformation is a very difficult question to answer, but log transformation is unlikely to be a problem for you here. You'll just need to remember that anything about that predictor will be reported on a log scale, and interpret accordingly.
$endgroup$
– Upper_Case
22 mins ago
$begingroup$
@user214 Left-skewed price information? That sounds interesting! (My research data is generally skewed hard to the right). There is always variation between study contexts, but I generally think of money as "geometric enough" that a log transformation is appropriate (or at least strongly defensible). Whether or not that's the ideal transformation is a very difficult question to answer, but log transformation is unlikely to be a problem for you here. You'll just need to remember that anything about that predictor will be reported on a log scale, and interpret accordingly.
$endgroup$
– Upper_Case
22 mins ago
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48711%2fwhat-are-the-disadvantages-of-having-a-left-skewed-distribution%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
