snippetpythonCriticalCanonical
How to deal with SettingWithCopyWarning in Pandas
Viewed 0 times
withhowpandassettingwithcopywarningdeal
Problem
I just upgraded my Pandas from 0.11 to 0.13.0rc1. Now, the application is popping out many new warnings. One of them like this:
What exactly does it mean? Do I need to change something?
How should I suspend the warning if I insist on using
The function that gives warnings
More warning messages
`E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
quote_df['TVol'] = quote_df['TVol']/TVOL_SCALE
E:\FinReporter\FM_EXT.py:450: SettingWithCopyWarning: A value is try
E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
quote_df['TVol'] = quote_df['TVol']/TVOL_SCALE
What exactly does it mean? Do I need to change something?
How should I suspend the warning if I insist on using
quote_df['TVol'] = quote_df['TVol']/TVOL_SCALE?The function that gives warnings
def _decode_stock_quote(list_of_150_stk_str):
"""decode the webpage and return dataframe"""
from cStringIO import StringIO
str_of_all = "".join(list_of_150_stk_str)
quote_df = pd.read_csv(
StringIO(str_of_all),
sep=',',
names=list('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefg'))
#dtype={'A': object, 'B': object, 'C': np.float64}
quote_df.rename(
columns={
'A':'STK', 'B':'TOpen', 'C':'TPCLOSE', 'D':'TPrice',
'E':'THigh', 'F':'TLow', 'I':'TVol', 'J':'TAmt',
'e':'TDate', 'f':'TTime'},
inplace=True)
quote_df = quote_df.ix[:,[0,3,2,1,4,5,8,9,30,31]]
quote_df['TClose'] = quote_df['TPrice']
quote_df['RT'] = 100 * (quote_df['TPrice']/quote_df['TPCLOSE'] - 1)
quote_df['TVol'] = quote_df['TVol']/TVOL_SCALE
quote_df['TAmt'] = quote_df['TAmt']/TAMT_SCALE
quote_df['STK_ID'] = quote_df['STK'].str.slice(13,19)
quote_df['STK_Name'] = quote_df['STK'].str.slice(21,30)#.decode('gb2312')
quote_df['TDate'] = quote_df.TDate.map(lambda x: x[0:4]+x[5:7]+x[8:10])
return quote_dfMore warning messages
`E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
quote_df['TVol'] = quote_df['TVol']/TVOL_SCALE
E:\FinReporter\FM_EXT.py:450: SettingWithCopyWarning: A value is try
Solution
The
The warning offers a suggestion to rewrite as follows:
However, this doesn't fit your usage, which is equivalent to:
While it's clear that you don't care about writes making it back to the original frame (since you are overwriting the reference to it), unfortunately this pattern cannot be differentiated from the first chained assignment example. Hence the (false positive) warning. The potential for false positives is addressed in the docs on indexing, if you'd like to read further. You can safely disable this new warning with the following assignment.
Other Resources
SettingWithCopyWarning was created to flag potentially confusing "chained" assignments, such as the following, which does not always work as expected, particularly when the first selection returns a copy. [see GH5390 and GH5597 for background discussion.]df[df['A'] > 2]['B'] = new_val # new_val not set in dfThe warning offers a suggestion to rewrite as follows:
df.loc[df['A'] > 2, 'B'] = new_valHowever, this doesn't fit your usage, which is equivalent to:
df = df[df['A'] > 2]
df['B'] = new_valWhile it's clear that you don't care about writes making it back to the original frame (since you are overwriting the reference to it), unfortunately this pattern cannot be differentiated from the first chained assignment example. Hence the (false positive) warning. The potential for false positives is addressed in the docs on indexing, if you'd like to read further. You can safely disable this new warning with the following assignment.
import pandas as pd
pd.options.mode.chained_assignment = None # default='warn'Other Resources
- pandas User Guide: Indexing and selecting data
- Python Data Science Handbook: Data Indexing and Selection
- Real Python: SettingWithCopyWarning in Pandas: Views vs Copies
- Dataquest: SettingwithCopyWarning: How to Fix This Warning in Pandas
- Towards Data Science: Explaining the SettingWithCopyWarning in pandas
Code Snippets
df[df['A'] > 2]['B'] = new_val # new_val not set in dfdf.loc[df['A'] > 2, 'B'] = new_valdf = df[df['A'] > 2]
df['B'] = new_valimport pandas as pd
pd.options.mode.chained_assignment = None # default='warn'Context
Stack Overflow Q#20625582, score: 1730
Revisions (0)
No revisions yet.